RE: how can we improve the performance of the job whil...
Minimize the transformer state Reference table have huge amount of date then you can use join stage. Reference table have less amount of data then you can use lookup.
RE: how can we improve the performance of the job while handling huge amount of data
this require a job level tuning or server level tuning. in job level we can do the follwing. job level tuning use Join for huge amount of data rather than lookup. use modify stage rather than transformer for simple transformation. Sort the data before remove duplicate stage.
server level tuning this can only be done after having adequate knowledge of the serever level parameter which can improve the server execution performance.
RE: how can we improve the performance of the job while handling huge amount of data
1. decrease usage of transformer stage.
2. use sorted input for aggregator.
3. Filter the data in source itself(if it is oracle database write user def sql)
4. use tunables tab in Administrator for cache memory buffer size ...( if u have knowledge-- if u have knowledge then only u go for that otherwise u will get some problems like deadlocks .....)
5. According to the requirement u use join lkp or merge(depends upon the data in input dataset and reference link data) use dataset for ref... link( for lookup)
6. use partition methods for different stages.
like that there are some performance techniqes are there.
