Load 10 Input files into 10 Target tables at a time

I have the 10 input file like F1,F2...F10 then I need to load these 10 input files into 10 target Output tables like T1,T2...T10. Here is the scenario for 10 tables But in future If i will receive 100 input files then I need to load it into respective 100 target tables.
After loading the input files into target tables I need a confirmation in the respective target tables(By input File name)
Please suggest me here how can I design this job in right way.

Questions by Aloka   answers by Aloka

Showing Answers 1 - 18 of 18 Answers

lavanya

  • Dec 29th, 2016
 

We have many ways to achieve this one of them is
Create sequence job as

• Write a routine to count the no of files and read the individual filename

• Use looping activity --to perform no of iterations
No of iterations=count/5

• Next use 5 job activities by calling the same parallel job in all job activities with different parameter values

For example:
If source file folder has 10 files
No of iterations =10/5=2 iterations
the loop perform 2 iterations each time 5 instances total 10 files will be processed.
If source folder has 100 files
No of iterations=100/5=20 iterations
The loop performs 20 iterations to execute 100 instances by processing 100 files.

Routine passes the count to loop activity ex 20

  Was this answer useful?  Yes

Ram

  • Jan 10th, 2017
 

Hi Aloka,
we can achieve this if you know the shell scripting. First design a job Source -->Target. then write a shell script for dsjob command to run the job and pass the parameters using parameter file for Source,schema & target tables. and run the job in loop.

  Was this answer useful?  Yes

Sirisha d

  • Feb 13th, 2017
 

In the below logic, I am making sure the input file names have similar name format like Input1,Input2,Input3.... to parameterize input file names .
Step 1 : Place all the input files in a single txt file with on file name in each line.
Make sure there is a new line at the end of all the input file names.
Step 2 : Open a new sequence job.
In the Execute command stage , give the following command.
wc -l <#file_path#
Step 3: In the User Varaiable Stage, create a new variable to hold the output of execute Command stage.
Make sure you are removing Field Marks.
Use following expression.
Convert(@FM,,Execute_Command_0.$CommandOutput)
Step 4: Take a start loop and define starting values as 1, Step =1 and To =#UserVariables_Activity.New#
Step 5 : Pass this Start loop counter value to a parallel job (in which logic is developed to load target files) .
The loop will be repeated as many times as the number of number of input files defined in the single input text file.
Each time a target file will be created for each input file.
Parellel Job : In the parallel job to load different text files, make sure the input file names and output file names are parameterized to accept value from the start loop counter.
If the input files have different formats or if we want to load tables instead of files, we can include more logic to the above design to achieve the result.
Hope this helps.

  Was this answer useful?  Yes

shasank

  • Apr 24th, 2017
 

We can achieve that by allowing multiple instances in a single job.
Just design a job that loads sources to target file and enable the multiple instances for that job.

I don't know exactly how to do it. As I have no Datastage software with. Try using that.

  Was this answer useful?  Yes

shasank

  • Apr 24th, 2017
 

We can do that allowing multiple instances for a single job in the job properties tab.

  Was this answer useful?  Yes

Reddy

  • Jun 27th, 2017
 

This can be achieved by creating multiple instance job and parameterizing the source file name, table name and also need to enable run time column propagation.

  Was this answer useful?  Yes

Give your answer:

If you think the above answer is not correct, Please select a reason and add your answer below.

 

Related Answered Questions

 

Related Open Questions