Loading...
Flags potential duplicate customer records by identifying rows with the same legal name, same date of birth, and similar address within the customer base. Duplicate records can lead to fragmented transaction monitoring, missed suspicious activity, and regulatory reporting failures.
Validates that a column qualifies as a valid primary key by ensuring all values are both unique and not null. Combines uniqueness and completeness checks into a single rule.
Detects near-duplicate records using string similarity measures. Identifies rows where key columns have similar but not identical values, such as name variations ("Jon Smith" vs "John Smith"), address typos, or inconsistent formatting.
Validates that the combination of specified columns forms a unique key. Ensures no two rows share the same values across all listed columns, enforcing composite key integrity.
Validates that values in a column are unique within each group defined by a partition column. For example, email must be unique per tenant, or employee_id must be unique per department.
Detects and counts duplicate rows based on specified columns. Returns the number of duplicates found and identifies the offending rows. Supports threshold-based alerting for acceptable duplicate rates.
Validates that no record IDs appear in both the training and test datasets. Data leakage between training and test sets leads to artificially inflated model performance metrics and unreliable AI systems. Under EU AI Act Article 10, datasets must support proper evaluation of AI system performance.
Validates that an auto-increment ID column has no gaps or duplicate values. Detects missing sequence numbers and repeated IDs that may indicate data loss, failed inserts, or replication issues.
Ensures no two rows are completely identical across all columns. Detects full-row duplicates that may indicate data loading issues, ETL bugs, or missing deduplication steps.
Validates that all non-null values in a specified column are unique. Useful for natural keys, email addresses, identifiers, and any column where duplicates indicate a data quality issue.
Business key must be unique in the target after load — a re-run or fan-out join often introduces duplicates.
Each resolved entity must have exactly one golden/master record — duplicate master_ids defeat the purpose of MDM.
Detect probable duplicate entities using normalized name + email/address/phone similarity above a threshold. Surfaces unmerged duplicates.
Where a 1:1 relationship is expected (e.g., user -> profile), the foreign key must be unique — no parent may have two children.
The local record id (jurisdiction + local case id) must be unique; duplicate notifications double-count cases in national surveillance.
The Universal Loan Identifier must uniquely identify each loan/application in the register. Duplicate ULIs trigger a validity edit.
The same concept reported for the same context and unit must not appear twice with different values (inconsistent duplicates).
A patient + condition + specimen should produce a single case report. Duplicates inflate counts and trigger redundant investigations.