SSIS Sort and Script To Remove Duplicates
Hello! I have developed a Package that will read from an Excel file, and then Sort the data on CaseID and then I use a Script Component to Remove Duplicates and send Unique Values to one DB Table and Duplicates to another. When I Sort the Data I get the following with my Duplicate Rows. What I would like to ask is how can I get the NULLS last in the Sort and my Row with Data First in the Sort? Everything is working but it is putting the Rows containing NULLS in my Valid Data Table instead of my Duplicate table. Thanks! Mike CaseID Current Tax Service Tax Service1 Tax Service2 Tax Service3 Transmittal Included on Address City County State Zip Code Acquisition AM NAID AM Name AM Assigned 271-936124 NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL 271-936124 2-Appraisal Complete Completed 07/06/2011 07/15/2011 07/20/2011 08/05/2011 Yes 401 ROSE DRIVE FARIBAULT RICE MN 55021-0000 06/29/2011 BSTSST9432 BEST ASSETS INC 07/05/2011Mike Kiser
July 17th, 2012 5:17pm

Hi Mike, There are two ways of doing it: 1) In your logic of segregating the unique and duplicate records, ignore any record with all nulls so that it is not considered at the time of doing the logic and only the 2nd row is considered as non duplicate and goes into valid table. Then in your logic of moving data into invalid (duplicate) table, include any rows with all other column values null, so that you dont miss any records. It will be either in Valid or invalid table. 2) Replace all NULLs with "ZZZ" in the previous step of sorting and segregating. This can be done with a derived column task. This should help as well. Later when you put into the table you can replace all "ZZZ" with NULLs again. Let me know if this doesn't help. Regards, Vinaya
Free Windows Admin Tool Kit Click here and download it now
July 17th, 2012 6:55pm

Vinaya, Option 1 worked!! Thanks! I really apprecaite the quick response! MikeMike Kiser
July 18th, 2012 11:33am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics