How can we improve the performance of the job while handling huge amount of data

Showing Answers 1 - 9 of 9 Answers

ramasamba

  • Nov 22nd, 2007
 

Minimize the transformer state,Reference table have huge amount of date then you can use join stage. Reference table have less amount of data then you can use lookup.

  Was this answer useful?  Yes

sunayan

  • Dec 13th, 2007
 

this require a job level tuning or server level tuning.
in job level we can do the follwing.

job level tuning

use Join for huge amount of data rather than lookup.
use modify stage rather than transformer for simple transformation.
Sort the data before remove duplicate stage.

server level tuning
this can only be done after having adequate knowledge of the serever level parameter which can improve the server execution performance.

  Was this answer useful?  Yes

1. decrease usage of transformer stage.

2. use sorted input for aggregator.

3. Filter the data in source itself(if it is oracle database, write user def sql)

4. use tunables tab in Administrator for cache memory, buffer size,...( if u have knowledge-- if u have knowledge, then only u go for that, otherwise u will get some problems like deadlocks,.....)

5. According to the requirement u use join, lkp or merge(depends upon the data in input dataset and reference link data), use dataset for ref... link( for lookup)

6. use partition methods for different stages.

like that there are some performance techniqes are there.

  Was this answer useful?  Yes

Give your answer:

If you think the above answer is not correct, Please select a reason and add your answer below.

 

Related Answered Questions

 

Related Open Questions