RE: How do you remove duplicate records from a file an...
Other than remove duplicate stage we can also use aggregator stage to count the number of records exist for the key columns.If more than one record exist for the key column then they are considered as duplicate records and using transformer we can set a stage variable as 'COUNT' and check if 'COUNT>1'.If so using a constraint reject that duplicate records into reject file.
RE: How do you remove duplicate records from a file and capture the rejected records in a separate file.
sort the records using the key field.In sort stage put "key change column true".Then zero will be assigned to the duplicate records.then put a condition as which is record is zero then send it to reject link
