Eliminate Duplicate Records

Hi

I am having 10000 records in flat file, in that there are 100 records are duplicate records..
soo we want to eliminate those records..which is the best method we have to follow.

Regards
Mahesh Reddy

Showing Answers 1 - 12 of 12 Answers

ravi

  • Oct 23rd, 2007
 

You can put sorter transformation after source qualifier transformation and in sorter tranformation' properties enable distinct property.

Thanks

kumar

vivek1708

  • May 11th, 2008
 

In order to beable to delete those entries
I think, you'll have to write sql queries in teh data base table
using rownum/rowid concept.

Or

by using the sorter and distinct option, load the unique rows in a temp table
followed by a truncate on the original table
and moving data back to it from the temp table.

hope it helps.

Using sorter (with distinct option checked) will not tell what all records were duplicate. There is even a better way of doing it without using aggregator and all. it just uses expression transformation.

  Was this answer useful?  Yes

Give your answer:

If you think the above answer is not correct, Please select a reason and add your answer below.

 

Related Answered Questions

 

Related Open Questions