1

Input

A file called input_file.csv, which has 7 columns, and n rows.

Example header and row:

Date Location Team1 Team2 Time Prize_$ Sport
2016 NY Raptors Gators 12pm $500 Soccer

Output

  1. n files, where the rows in each new file are grouped based on their values in column 7 of the original file. Each file is named after that shared value from column 7. Note: each file will have the same header. (The script currently does this.)

    Example: if 2 rows in the original file had golf as their value for column 7, they would be grouped together in a file called golf.csv. If 3 other rows shared soccer as their value for column 7, they would be found in soccer.csv.

  2. An array that has the name of each generated file in it. This array lives outside of the scope of awk. (This is what I need help with.)

    Example: Array = [golf.csv, soccer.csv]

Situation

The following script produces the desired output. However, I want to run another script on each of the newly generated files and I don't know how.

Question:

My idea is to store the names of each new file in an array. That way, I can loop through the array and do what I want to each file. The code below passes a variable called array into awk, but I don't know how to add the name of each file to the array.

#!/bin/bash

ARRAY=()

awk -v myarray="$ARRAY" -F"\",\"" 'NR==1 {header=$0}; NF>1 && NR>1 {if(! files[$7]) {print header >> ("" $7 ".csv"); files[$7]=1}; print $0 >> ("" $7 ".csv"); close("" $7 ".csv");}' input_file.csv

for i in "${ARRAY[@]}"
    do
    :
    echo $i
done
Improve this question
4
  • The linked answer doesn't explain how to add each filename to an array. I tried exporting to a file, but none of the filenames are being stored anywhere. If I knew how to add each filename to an array, I think I could figure out how to access that array outside of awk. Commented Feb 26, 2016 at 21:34
  • How do I store the name of each file - what file(s)? If you can provide a better explanation and concise, testable sample input and expected output I for one would consider voting to reopen but as it stands it looks like the question yours is closed as a dup of DOES contain the answer to your question. Commented Feb 26, 2016 at 21:39
  • @EdMorton Is this edit clearer? Commented Feb 27, 2016 at 0:36
  • Yes but I don't understand why you'd post a space-separated input file when you say your real one is comma-separated nor why you didn't create an input file with say a couple more lines and the output files you;d want generated from that input file to make it 100% clear. Oh well I think I know what you want now. Commented Feb 27, 2016 at 13:38

5 Answers 5

Reset to default
2

Rather than struggling to get awk to fill your shell array variable, why not:

  • make sure that the *.csv files are created in a clean directory
  • use globbing to loop over all *.csv files in that directory?
awk -F'","' ...  # your original Awk command

for i in *.csv  # use globbing to loop over resulting *.csv files
    do
    :
    echo $i
done
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

you can even put the files in an array afterward if you still want to do that: filesArray=(*.csv)
Ya, that makes great sense. But this doesn't address the challenges the OP described in the question, does it?
@ghoti: This answer suggests an alternative way to approach the problem (and is hopefully clear in that regard), because I suspect the question to be an instance of the XY problem.
I agree that this is an XY problem -- but then, 80% of the questions we answer here probably are, if you look deep enough. Why does the OP want separate files by $7? What's the purpose of the array? I'm certain we could figure out a better way to achieve his goals. But I didn't see those goals in his question.
0

Just off the top of my head, untested because you haven't supplied very much sample data, what about this?

#!/usr/bin/awk -f

FNR==1 {
  header=$0
  next
}

! $7 in files {
  files[$7]=sprintf("sport-%s.csv", $7)
  print header > file
}

{
  files[$7]=sprintf("sport-%s.csv", $7)
}

{
  print > files[$7]
}

END {
  printf("declare -a sportlist=( ")
  for (sport in files) {
    printf("\"%s\"", sport)
  }
  printf(" )\n");
}

The idea here is that we store sport names in the array files[], and build filenames out of that array. (You can format the filename inside sprintf() as you see fit.) We step through the file, adding a header line whenever we get a new sport with no recorded filename. Then for non-headers, print to the file based on the sport name.

For your second issue, exporting the array back to something outside of awk, the END block here will output a declare line which can be interpreted by bash. IF you feel lucky, you can eval this awk script inside command expansion, and the declare command will effectively be interpreted by your shell:

eval $(/path/to/awkscript inputfile.csv)

Or, if you subscribe to the school of thought that consiers eval to be evil, you can redirect the awk script's standard output to a temporary file which you source:

/path/to/awkscript inputfile.csv > /tmp/yadda.$$
. /tmp/yadda.$$

(Don't use this temp file, make a real one with mktemp or the like.)

Improve this answer

Comments

0

There's no way for any program to modify the environment of the parent shell. Just have the awk script output the names of the files as standard output, and use command substitution to put them in an array.

filesArray=($(awk ... ))

If the files might have spaces in them, you need a different solution; assuming you're on bash 4, you can just be sure to print each file on a separate line and use readarray:

readarray filesArray < <( awk ... )

if the files might have newlines in them, too, then things get tricky...

Improve this answer

Comments

0

if your file is not large, you can run another script to get the unique $7 elements, for example

$ awk 'NR>1&&!a[$7]++{print $7}' sports

will print the values, you can change it to your file name format as well, such as

$ awk 'NR>1&&!a[$7]++{print tolower($7)".csv"}' sports

this then can be piped to your other process, here for example to wc

$ awk ... sports | xargs wc
Improve this answer

Comments

0

This will do what I THINK you want:

oIFS="$IFS"; IFS=$'\n'
array=( $(awk '{out=$7".csv"; print > out} !seen[out]++{print out}' input_file.csv) )
IFS="$oIFS"

If your input file really is comma-separated instead of space-separated as you show in the sample input in your question then adjust the awk script to suit (You might want to look at GNU awk and FPAT).

If you don't have GNU awk then you'll need to add a bit more code to close the open output files as you go.

The above will fail if you have file names that contain newlines but will be fine for blank chars or other white space.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.