The Duplicates can be eliminated by loading thecorresponding data in the Hash file. Specify the columns on which u want to eliminate as the keys of hash.
removal of duplicates done in two ways: 1. Use Duplicate Data Removal stage or 2. use group by on all the columns used in select duplicates will go away.
sql: delete from tablename a where rowid>(select min(rowid) from tablename b) where a'key values b.key values. datastage server:using sort stage(option:allow duplicates(yes/no)) or hash file
