Eliminating Header and Trailor without using next in sequence and Dedup sort

I have header and trailer records to some body records like below
Here there is no record indicator.
How will you eliminate and separate header and trailer records without using Next In Sequence, Dedup sorted and any Partition components?

Showing Answers 1 - 18 of 18 Answers


  • Aug 2nd, 2016

use rollup component.

  Was this answer useful?  Yes


  • Aug 31st, 2016

Could you please explain how rollup can be work for this?

  Was this answer useful?  Yes

Anil modala

  • Sep 18th, 2016

In the roll-up by using last and first functions, You can achieve this.

  Was this answer useful?  Yes


  • Sep 26th, 2016

Use filter by expression

  Was this answer useful?  Yes


  • Oct 1st, 2016

use two dedup
1> dedup keep first
2> keep last
3> key {}

  Was this answer useful?  Yes

Prakash Poongavanam

  • Oct 13th, 2016

In case of no indicator how will you read this file? Conditional DML is possible only if there is record_change indicator.

  Was this answer useful?  Yes


  • Oct 14th, 2016

Use Rollup component in expanded mode and select the first and last records.

  Was this answer useful?  Yes


  • Oct 19th, 2016

Reformat should do this; using o/p index or indexes

  Was this answer useful?  Yes


  • Feb 28th, 2017

How to use Filter By expression?

  Was this answer useful?  Yes


  • Mar 17th, 2017

Header-Tailer Without Indicator

1.Input file -->Filter by expression (next_in_sequence()>=1) we can collect header record at outport and de select port: We will get the body+data.

2.Connect to Dedupe keep key as null and keep last, Now we will get the count value to outport

3.Body to dupe port

Input--->Filter by expression --->De dupesort--->Outport

  Was this answer useful?  Yes


  • Mar 20th, 2017

for the above file first write input dml as string("
") newline;
and then the data will be stored in input file
and the connect dedup sort port and one out port of it to a file and other out port of it to a dedup sort in port and do the sort using null key and
and then in the keep format place as first and then the first record will goes means header will go and stays in a file and the left content will goes to second dedup sort and in this same go with null key and this time keep parameter as last then last will goes and stay in a file connected to it and rest the data will goes into other file
input ---- dedup dort... 1) op file2
2)dedupsort ..................1)op file2

  Was this answer useful?  Yes


  • Mar 28th, 2017

Read the file using DML-->string(
) emp_details;
Add filter by expression and put a condition--> length_of(string_split(emp_details,/))==3.
It will filter out the header and trailer

  Was this answer useful?  Yes

eswara kalyan

  • Mar 29th, 2017

We can achieve this in multiple ways:

Use a dedup with null key and using the option keep first or keep last u can eliminate the first or last record.

Using a rollup you can use the function first (in.data) so that you get the first record from the file. Similarly last (in.data) gets last record

  Was this answer useful?  Yes


  • Jun 6th, 2017

1. Read the records as a single field.
2. Replicate the data, and connect one flow to Leading Records component. Keep the value as 1, so that it outputs only header record.
3. Join output of step 2 to second flow of replicated data. Connect the output to trash, now header alone is eliminated.
4. Now replicate the data of unused port1 of join used in step3.
5. Connect one flow to Rollup, with null key, and leave input must be sorted as it is. Rollup will return the last record.
6. Join output of step 5 to second flow of replicated data from step 4. Connect the output to trash, now trailer alone is eliminated.
7. Unused port1 dat from Join will give only the data records.

  Was this answer useful?  Yes


  • Aug 22nd, 2017

Using rollup component, pass null key, then the select port contain the last record, the op of deselect port used as a lookup use reformat component and use output index here we use lookup count function to achieve the scenario, else use first and last function in rollup

  Was this answer useful?  Yes


  • Dec 10th, 2018

1) Add Sequence to each line.
Input file: your file having input
Output file with sequence added : out1
2) Create file having first rec
sequence number position equals 1
Input : Ypur input file
Output: file having header record : out2
3) Add trailer record to file
3) Icetool
In : Your input file
Subst from {in} to {out} keep input last
Output : file having header and trailer record:out2
4)Copy non matching records using join keys
Infile1 : input file
Infile2: outfile
Output: file without header and trailer record : out3

  Was this answer useful?  Yes

Ketan Khot

  • Apr 12th, 2019

We can achive this with only one reformat component.
graph can be -
Input FIle --> Reformat (with output index)--> output files (1-header, 2-data, 3-trailer)
write below code to separate out header/ data/ trailer records in output index
type header_t =
string("|") filename;
") date;
type detail_t =
string("/") f1;
string("/") f2;

output_index_out :: output_index(in) =
output_index_out :: if(not is_null(reinterpret_as(header_t,in.line))) 0 //if reinterpret_as function return not null that means input record is header record
else if(not is_null(reinterpret_as(detail_t,in.line))) 1 //if reinterpret_as function return not null that means input record is detail record
else 1
note : reinterpret_as function return null vallue when input data is not formated in given dml

  Was this answer useful?  Yes


  • Dec 3rd, 2019

Use scan to take count(). In output select filter count = 0 and max(count()) which could be got by using a global variable in scan to save the max value.

  Was this answer useful?  Yes

Give your answer:

If you think the above answer is not correct, Please select a reason and add your answer below.


Related Answered Questions


Related Open Questions