Our SSIS component SQLPhonetics.NET for SQL Server Integration Services allows you to identify duplicates in large databases within your ETL process.
The components allow you to create search environments with different phonetic and distance algorithms. You can not only check individual columns for duplicates, but also identify them in extensive datasets with several columns.
The various parameters of the component also allow you to classify the weighting of hits, perform cross comparisons across several columns, directly exclude NULL values or perform a partial comparison with data from different sources.
The component can check several million data records in less than one hour for duplicates within your data flow.
The component exists for SQL Server 2012 – 2017 and can also be used in the Azure SSIS IR.