-
Datastage scenario Question
A Sequences is calling activity 1, activity 2 and activity 3.while running, activity 1 and 2 got finished but 3 got aborted. How can I design a sequence such that the sequence has to run from activity 2 when I restart the sequences?
Mallikarjuna_G
- May 8th, 2014
To make the job re-run from activity 3, we need to introduce restartability in the sequence job. For this below points have to be taken care of in Job Sequence Adding Checkpoints: Checkpoints have t...
Ritwik
- Apr 21st, 2014
You have to check the " Do not checkpoint run " checkbox for activity 2. If you set the checkbox for a job that job will be run if any of the job later in the sequence fails and the sequence is restarted.
-
Datastage Performance Tuning
How to improve the Performance Tuning any 5 points
Lalit
- Oct 6th, 2017
1)Set Proper configuration file
2)Handle null and duplicate values properly
3)Reduce number of transformers as much as possible
4)Do not take more than 20 stages in a perticuler job
5)Use Dataset stage instead of sequential file stage
6)Use join merge and lookup stages properlyMallikarjuna_G
- Aug 10th, 2013
Use Copy stage instead of a Transformer for simple operations like : •placeholder between stages •renaming Columns •dropping Columns •implicit (default) type Conversions USe Stage variab...
-
Count in Dataset
How to get the dataset record count? with out using orchadmin command
Priyanka Sinha
- Mar 17th, 2016
1) The Datastage Designer GUI (also available Manager and Director) provides a mechanism to view and manage data sets. It can be invoked in Tools -> Data set management. 2) orchadmin command-line uti...
Mallikarjuna_G
- Aug 10th, 2013
There are only two ways to this -
1. Use datastage dataset management utility[GUI].
2. Use orchadmin utility from commandline -
Job running slow
Hi, Job1 is running for 10 minutes for the first time and the same job1 is running for 15 minutes for the second time ,the load being the same for both.Could someone explain..
Mallikarjuna_G
- Aug 10th, 2013
One reason could be that - for the first time the job is just loading/inserting data into target. When you run the same job again, it would take more time as it tries to update from second time. Upda...
Raju Nath
- Jun 27th, 2013
Hi you can use the delete and then load command during loading then it will be the same time. First time when you are loading that time the dataset/table is empty that is why its taking less time bu...
-
No of processors in DataStage
How do we know the no of processors in a job? Is there any specific calculation for this.
Mallikarjuna_G
- Aug 10th, 2013
There will not be any processors in the job. Your question could be - find the number of nodes/processors on which the job is running. So the answer to this - go to Director and open the log for the ...
Kumaresh
- Jul 19th, 2013
Run the job. Go to director log. Check for the apt config data displayed over there. That will show the number of processors/nodes.
-
Which partition is best in performance level?
Mallikarjuna_G
- Aug 10th, 2013
It depends on the job design and requrement. When you specify SAME partitioning, datastage uses the partioing method defined in the previous stage and will not perform any partitioning in the current...
Muralidhar
- May 30th, 2013
Same partition.
The reason is it keeps the previous partition and send to output as it is. -
Datastage partition
Which partition followed by joiner,merge. lookup,remove duplicate
lalit
- Oct 9th, 2017
For join, merge and remove duplicated you can use Hash partitioning and for lookup you can use entire partitioning
Thank you
Mallikarjuna_G
- Aug 10th, 2013
For Join, Merge and Remove duplicates, have data on links hash key partitioned an sorted on Key columns specified. For lookup - primary link needs to be hash key partitioned and sorted and reference link has to use entire partition method.
-
How do you use Surrogatekey key in reporting?
Mallikarjuna_G
- Aug 10th, 2013
In a typical data warehouse environment, we normally have a requirement to keep history. So, we would end up having multiple rows for a given Primary key. So, we define a new column that doesn't have ...
djkool4u
- Jul 11th, 2013
Is Slowly Changing Dimension (SCD) Implementation we use Surrogate key as the primary key is being duplicated for the sake of keeping history data for the records with the same pk.
-
-
what is the purpose of the copy stage:
what is the uses of the copy stage with out copies input link to output link datasetsand it can have any purposes pls send me with example
Mallikarjuna_G
- Aug 10th, 2013
Use Copy stage for simple operations like :
•to multicast the same input data among multiple output links.
•placeholder between stages to avoid buffer issues
•renaming Columns
•dropping Columns
•implicit (default) type ConversionsAkila
- Apr 1st, 2013
Besides used for making copies of the input, copy stage is also helpful for achieving 1) dropping columns between stages 2) to change column names 3) if the scenario is to end the job flow directly f...
-
fetching last row from a particular column of sequential file
how to fetch the last row from a particular column.. Input file may be sequential file...
Mallikarjuna_G
- Aug 10th, 2013
There are multiple ways to do this, given that input is a sequential file. Two of them are
1. Use "Filter" option available in Seq file stage. Specify a unix command like: tail -1 (recommended)
2. Use a tail stage next to Seq filestage and then specify.Akil
- Apr 1st, 2013
There are 2 ways to fetch the last row of any file if the number of records in the file is not known before run time:
1) Use the tail stage and run the tail stage in sequential mode
2) Use the lastrow() function in the transformer stage -
How to connect two stages which do not have any common columns between them?
Mallikarjuna_G
- Aug 10th, 2013
If those two stages are sources and if it is valid that though the names of columns from two sources differ but they are same, then use a copy stage infront of one source stage and rename column as pe...
muralidhar
- May 30th, 2013
This is little tricky question.we can do in two ways.
1)we can connect using lookup stage if both columns having same datatype.
2)We can add one extra column that is common for two files then we can connect. -
What are Sequencers?
Sequencers are job control programs that execute other jobs with preset Job parameters.
Read Best Answer
Editorial / Best Answer
jagannimmagadda- Member Since Jan-2006 | Jan 10th, 2006
A sequencer allows you to synchronize the control flow of multiple activities in a job sequence. It can have multiple input triggers as well as multiple output triggers.The sequencer operates in two modes:ALL mode. In this mode all of the inputs to the sequencer must be TRUE for any of the sequencer outputs to fire.ANY mode. In this mode, output triggers can be fired if any of the sequencer inputs are TRUE
Tariq Awan
- Oct 25th, 2016
As previous posts already pointed out, there are only two modes: All or Any. All means all incoming triggers must be true and Any means at least one needs to be true. However, in case, if you want to execute next job even though all the incoming triggers are False, then Seqencer is NOT the solution.
Priyanka
- Mar 10th, 2016
@Rohit: Yes you can run the 2nd job even if the first one fails. For that you need to set the trigger option to unconditional. However, the control comes to the 2nd job only after the 1st job has run (pass or fail).
-
How did you handle an 'Aborted' sequencer?
In almost all cases we have to delete the data inserted by this from DB manually and fix the job and then run the job again.
Read Best Answer
Editorial / Best Answer
ritu singhai- Member Since Jul-2008 | Jul 5th, 2008
Have you set the compilation options for the sequence so that in case job aborts, you need not to run it from from the first job. By selecting that compilation option you can run that aborted sequence from the point the sequence was aborted.
Like for example, you have 10 jobs(job1, job2, job3 etc.) in a sequence and the job 5 aborts, then by checking "Add checkpoints so sequence is restartable on failure" and "Automatically handle activities that fail" you can restart this sequence from job 5 only. it will not run the jobs 1,2,3 and 4.
Please check these options in your sequence.
Hope this helps.Mallikarjuna_G
- Aug 10th, 2013
Two things needs to be handled when a Job Sequence aborts. 1. We must have an exception handling code that notifies us about the failure 2. When we re-run the sequence after fixing the Sequence, it ...
Riten
- May 30th, 2012
To Handle Aborted Sequence These are the steps to be taken: First Add Terminator to Job Sequence and Choose Terminator with Other option(Trigger in Job sequence) ( if job fails it will go to Other lin...
