Abinitio - How do you improve the performance of a graph?

GeekInterview.com > Interview Questions > Data Warehousing > Abinitio

Go To First | Previous Question | Next Question

Abinitio | Question 6 of 196

How do you improve the performance of a graph?

Total Answers and Comments: 4

Last Update: May 10, 2006

Sponsored Links

Best Rated Answer

—

Submitted by: Ashim Dutta

ashimdutta@yahoo.com
There are many ways the performance of the graph can be improved.
1) Use a limited number of components in a particular phase
2) Use optimum value of max core values for sort and join components
3) Minimise the number of sort components
4) Minimise sorted join component and if possible replace them by in-memory join/hash join
5) Use only required fields in the sort, reformat, join components
6) Use phasing/flow buffers in case of merge, sorted joins
7) If the two inputs are huge then use sorted join, otherwise use hash join with proper driving port
8) For large dataset don't use broadcast as partitioner
9) Minimise the use of regular expression functions like re_index in the trasfer functions
10) Avoid repartitioning of data unnecessarily

Above answer was rated as good by the following members:
njvijay

July 10, 2005 11:28:15

Ashim Dutta

RE: How do you improve the performance of a graph?

ashimdutta@yahoo.com
There are many ways the performance of the graph can be improved.
1) Use a limited number of components in a particular phase
2) Use optimum value of max core values for sort and join components
3) Minimise the number of sort components
4) Minimise sorted join component and if possible replace them by in-memory join/hash join
5) Use only required fields in the sort reformat join components
6) Use phasing/flow buffers in case of merge sorted joins
7) If the two inputs are huge then use sorted join otherwise use hash join with proper driving port
8) For large dataset don't use broadcast as partitioner
9) Minimise the use of regular expression functions like re_index in the trasfer functions
10) Avoid repartitioning of data unnecessarily

Is this answer useful? Yes | No

1 0

Overall Rating: +1

September 13, 2005 05:11:16

mithun ganguly

RE: How do you improve the performance of a graph?

One of the vital points is that :

Try to run the graph as long as possible in MFS. For these input files should be partitioned and if possible output file should also be partitioned.

Is this answer useful? Yes | No

January 18, 2006 01:22:44

Gour

RE: How do you improve the performance of a graph?

is in-memory join faster than sorted join ??

Is this answer useful? Yes | No

May 10, 2006 01:54:05

Baanumathy

Member Since: May 2006 Contribution: 1

RE: How do you improve the performance of a graph?

In addition to the above mentioned cases we can also include some more like:

1. If sort component is used and the sort keys are same for the next sort component which follows after 2 or 3 components then instead of using sort component again it is preferable to use Sort within Groups component mentioning thise keys as major keys and other keys as minor keys. in this case it assumes that major keys are already sorted and it need sto sort only on minor keys.

eg: sort-1 component uses keys a b c and 2nd sort component after 2 - 3 components (in the same flow) uses keys a b e f. in that case use sort within groups in the 2nd case keeping a b as major keys and e f as minor keys.

2. when splitting records into more than two flows prefer Reformat rather than Broadcast component.

3. For joining records from 2 flows use Concatenate component only when there is a need to follow some specific order in joining records. If no order is required then it is preferable to use Gather component.

4. Instead of too many Reformat component consecutively one after the other use output indexes parameter in the first Reformat component and mention the condition there. For detailed information on this concept refer Help.

Is this answer useful? Yes | No

1 0

Overall Rating: +1

Go To Top

User Name		Remember Me?
Password

Question Bank