0

My file has multiple headers and multiple record types (eg. 0001, 0002, 0003, 0004). Count is given for each record type in the trailer row along with overall detail record count.

Sample File:

XYZH001
YZXH002
0001Rec1
0001Rec2
YZXH002
0002Rec1
0002Rec2
YZXH002
0003Rec1
0003Rec2
0003Rec3
YZXH002
0004Rec1
T999008002002004001

File details:

Detail records are where 1 to 4 position data in (0001, 0002, 0003, 0004)

Trailer:
Trailer identifier(position 1 to 4)             = T999
total data count (position 5 to 7)              = 008
count of record type 0001 (position 8 to 10)    = 002
count of record type 0002 (position 11 to 13)   = 002
count of record type 0003 (position 14 to 16)   = 004
count of record type 0004 (position 17 to 19)   = 001

Requirement:

-- Compare overall detail row count where 1 to 4 position data in (0001, 0002, 0003, 0004) with trailer record count (position 5 to 7) 
-- Compare each Record type row count with trailer record count
     eg. Compare row count where 1 to 4 position data = 0001 with trailer record count for 0001 (position 5 to 7) 
      .....
-- Stop execution in case of detail record row count and trailer count mismatch

Expected output :

Overall detail row count 8 matches with trailer record count 8.
Row count for 0001 record type 2 matches with trailer record count 2.
Row count for 0002 record type 2 matches with trailer record count 2.
Row count for 0003 record type 3 does not match with trailer record count 4.
Stopping execution.
Improve this question
3
  • @roaima Posted my approach in answer section Commented May 3, 2021 at 9:31
  • 1
    example code belongs in the question, same as your example data - not as an answer (unless you are answering your own question). Commented May 3, 2021 at 9:47
  • From what you posted as an answer, I can only recommend that you write the whole script in awk (or perl. or python) rather than running awk half a dozen or so times against the same data, plus another half a dozen or so calls to sed. What you're trying to do can (and should) be done in a single pass through the file, not multiple passes. Hint: perl has awk and sed like syntax built-in. and awk has sub() and gsub() functions which can do sed-like search & replace. Commented May 3, 2021 at 9:53

3 Answers 3

Reset to default
1

Using awk:

awk '/^000[1-4]/{ rec[substr($0, 1, 4)+0]++; totalRec++ }
END{ trailTotalRec=substr($0, 5, 3)+0
     for(i=1; i<=4; i++) { trailRec[i]=substr($0, pos+8, 3)+0; pos+=3 }
    if(trailTotalRec==totalRec)
        print "Overall detail row count", totalRec, "matches with trailer record count", trailTotalRec"."
    for(i=1; i<=4; i++) {
        print "Row count for 000" i, "record type", rec[i], (trailRec[i]==rec[i]?"matches":"does not"), "with trailer record count", trailRec[i]"."
        if(trailRec[i]!=rec[i]) { print "Stopping execution."; exit 7 }
    }
}' infile
Improve this answer
0
0 
#!/usr/bin/awk -f

/^000[0-9]/ {
    i=substr($1,1,4);
    sub(/^0+/,"",i);
    bc[i]++;   # bc = body count array
    rtotal++;
}

/^T999/ {
  dc = substr($0,5,3);
  for (r in bc) {
    rc[r] = substr($0,(r-1)*3+8,3); # rc = row count array
    sub(/^0+/,"",rc[r]);
    ttotal+=rc[r]
  };
}

END {
  if (dc == rtotal) {
    printf "Overall detail row count %i matches with trailer record count %i.\n", rtotal, dc;
  } else {
     print "Row count %i does not match trailer record count %i", rtotal, dc;
     exit 1;
  }
  for (r in rc) {
    if (rc[r] == bc[r]) {
      printf "Row count for %04i record type %i matches with trailer record count %i.\n", r, bc[r], rc[r];
    } else {
      printf "Row count for %04i record type %i does not match with trailer record count %i.\n", r, bc[r], rc[r];
      print "Stopping execution"
      exit 2;
    };
  };
}

This assumes that the T999 trailer line will have 3 characters for the data count (dc) AND for each row of the bc array.

If there are any errors in the data, it will exit with either a 1 (mismatch in row total vs trailer total data count) or a 2 (mismatch in row total vs trailer record). This can be tested and used in, e.g., a shell case statement with the shell built-in variable $? (the exit code for the last command):

 #!/bin/sh

 ./summarise.awk file.txt
 ec="$?"
 case "$ec" in
    0) probably_do_nothing ;; # success! 0 == no error
    1) do_something ;;
    2) do_another_thing ;;
 esac

Save as, e.g. summarise.awk, and make executable with chmod +x summarise.awk.

$ ./summarise.awk file.txt 
Overall detail row count 8 matches with trailer record count 8.
Row count for 0001 record type 2 matches with trailer record count 2.
Row count for 0002 record type 2 matches with trailer record count 2.
Row count for 0003 record type 3 does not match with trailer record count 4.
Stopping execution
Improve this answer
3
  • would i be able to achieve this in a single script. Actually i've to create a single script to validate header, trailer, file format. I am not able to merge this in my script, i am new to unix. Commented May 3, 2021 at 11:31
  • 1
    It is a single script. If you're asking "can i turn this awk script into a one-liner because I hate being able to read or re-use my own code" then, yes, the answer is you can always turn any awk script into a one-liner. ditto for perl. and sed. and several other scripting languages. BTW, if you're thinking "but it's an awk script, it's somehow magically different to running sed or grep or any of 10000 other programs from within a shell script" then you need to re-think that, because it's 100% wrong. Commented May 3, 2021 at 11:35
  • rule of thumb: Use one-liners for short, easily understood things. a few hundred characters at the very most. Use stand-alone scripts for anything longer or more complicated, or if editing it on the command line is a PITA. or if you want to avoid shell quoting hassles. or that you're likely to re-use for other tasks (i.e. same as you'd write a shell script for stuff you're likely to do repeatedly). Commented May 3, 2021 at 11:41
0 
$ cat tst.awk
/^[0-9]/ {
    currType = substr($0,1,4)
    if ( currType != prevType ) {
        recNr2type[++actNumRecs] = currType
        prevType = currType
    }
    recNrs2rowCnts[actNumRecs]++
    actTotRows++
    next
}
{ trailer = $0 }
END {
    expTotRows = substr(trailer,5,3)+0

    result = (actTotRows == expTotRows ? "matches" : "does not match")
    printf "Overall detail row count %d %s with trailer record count %d.\n", \
        actTotRows, result, expTotRows

    trailer = substr(trailer,8)
    while ( (trailer != "") && (result == "matches") ) {
        expRowCnt = substr(trailer,1,3)+0
        actRowCnt = recNrs2rowCnts[++expNumRecs]
        type      = recNr2type[expNumRecs]

        result = (actRowCnt == expRowCnt ? "matches" : "does not match")
        printf "Row count for %s record type %s %s with trailer record count %d.\n", \
            actRowCnt, type, result, expRowCnt

        trailer = substr(trailer,4)
    }

    print "Stopping execution."
}


$ awk -f tst.awk file
Overall detail row count 8 matches with trailer record count 8.
Row count for 2 record type 0001 matches with trailer record count 2.
Row count for 2 record type 0002 matches with trailer record count 2.
Row count for 3 record type 0003 does not match with trailer record count 4.
Stopping execution.
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.