DataStage - Remove duplicates using transformer

GeekInterview.com > Interview Questions > Data Warehousing > DataStage

Go To First | Previous Question | Next Question

DataStage | Question 368 of 390

Remove duplicates using transformer

how do you remove duplicates using transformer stage in datastage

Total Answers and Comments: 3

Last Update: October 26, 2009 Asked by: nagakalyan

Sponsored Links

Best Rated Answer

—

Submitted by: umamahesh.ds

In that Time double click on transformer stage---> Go to Stage properties(its having in hedder line first icon) ---->double click on stage properties --->Go to inputs ---->go to partitioning---->select one partition technick(with out auto)--->now enable perform sort--->click on perfom sort----> now enable unique---->click on that and we can take required colum name. now out put will come unique values so here duplicats will be removed.

Above answer was rated as good by the following members:
Tiger81

June 25, 2008 10:17:35

vira_venkat

Member Since: May 2008 Contribution: 1

RE: Remove duplicates using transformer

You have to put primary key on the column on which you want to remove duplicates.
That will remove the duplicates.If You want to catch the rejected rows you can apply ''rejected'constraint in the constraint tab of transformer.

Is this answer useful? Yes | No

0 1

Overall Rating: -1

July 25, 2008 02:42:30

umamahesh.ds

Member Since: July 2008 Contribution: 2

RE: Remove duplicates using transformer

In that Time double click on transformer stage---> Go to Stage properties(its having in hedder line first icon) ---->double click on stage properties --->Go to inputs ---->go to partitioning---->select one partition technick(with out auto)--->now enable perform sort--->click on perfom sort----> now enable unique---->click on that and we can take required colum name. now out put will come unique values so here duplicats will be removed.

Is this answer useful? Yes | No

1 0

Overall Rating: +1

October 26, 2009 04:12:27

Jaimy Chacko

Member Since: October 2009 Contribution: 1

RE: Remove duplicates using transformer

To capture rejected duplicates use a Transformer. Partition and sort on your primary key. In a transformer keep the primary key stored in a Stage Variable. Compare incoming primary key to the stored primary key Stage Variable. If it is the same output the incoming row as a duplicate if it is different output the row as unique and save the new primary key.

You need at least two stage variables one to do the comparison and the other to store the key value:

Variable: Derivation
IsDuplicate: input.keyfield SavedKey
SavedKey: input.keyfield

Special thanks to Vincent

Is this answer useful? Yes | No

Go To Top

User Name		Remember Me?
Password

Question Bank