BASH: append to array

Question

Please be aware that this is just a pseudo code to illustrate the problem:

Directory structure:

# tree
.
├── dir1
│   ├── d1_file1
│   ├── d1_file2
│   └── d1_file3
└── dir2
    ├── d2_file1
    ├── d2_file2
    └── d2_file3

2 directories, 6 files

Script:

#!/bin/bash

list=(
dir1
dir2
)

for dir in ${list[@]}; do
    ls -1 ./tmp/${dir} | while read -a file; do
    files+=(${file})
    echo "${dir} has ${#files[@]} files(s)"
    echo "${dir} files: ${files[@]}"
    done
done

Current output:

# bash tmp.sh 
dir1 has 1 files(s)
dir1 files: d1_file1
dir1 has 2 files(s)
dir1 files: d1_file1 d1_file2
dir1 has 3 files(s)
dir1 files: d1_file1 d1_file2 d1_file3
dir2 has 1 files(s)
dir2 files: d2_file1
dir2 has 2 files(s)
dir2 files: d2_file1 d2_file2
dir2 has 3 files(s)
dir2 files: d2_file1 d2_file2 d2_file3

Desired outcome:

# bash tmp.sh
dir1 has 3 files(s)
dir1 files: d1_file1 d1_file2 d1_file3
dir2 has 3 files(s)
dir2 files: d2_file1 d2_file2 d2_file3

Basically I would like to build a separate array for each item in the for loop that stores values from the ls command.

Tom Fenech · Accepted Answer · 2015-05-10 14:34:55Z

2

In order to only output the count once per directory, you need to move the printing out of the while read loop. However, I would suggest removing it entirely. Try something like this:

#!/bin/bash

list=(
dir1
dir2
)

for dir in "${list[@]}"; do
    files=( ./tmp/"$dir"/* )
    echo "${dir} has ${#files[@]} files(s)"
    echo "${dir} files: ${files[@]}"
done

This uses a glob to populate the files array with the contents of each directory.

In general, parsing ls should be avoided.

answered May 10, 2015 at 14:34

Tom Fenech

75.1k13 gold badges119 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

HTF Over a year ago

It looks like files variable is empty when I move the print out of the while loop?

Tom Fenech Over a year ago

@HTF that's because piping creates a sub shell, so variables declared inside are lost. There are ways around this but I would suggest avoiding the problem entirely and using the approach I have suggested in my answer.

HTF Over a year ago

Thanks, please note that this is just a pseudo code, basically I didn't want to paste the whole script so I just used ls for example. The original script is used to iterate through AWS S3 buckets.

Tom Fenech Over a year ago

@HTF fair enough, in that case one option would be to pipe the other aws command to a block { while read; do ... done; echo ... }

HTF Over a year ago

Thanks, reference is someone else will need this info: mywiki.wooledge.org/BashFAQ/024

mklement0 · Accepted Answer · 2015-05-10 14:40:37Z

1

You can simply use pathname expansion (globbing) to capture all the files in your subdirectories:

list=(
 dir1
 dir2
)

for dir in "${list[@]}"; do
    files=( ./tmp/"${dir}"/* )  # collect all matches in array
    echo "${dir} has ${#files[@]} files(s)"
    echo "${dir} files: ${files[@]}"
done

Note how I've double-quoted ${list[@]} and ${dir} to protect their values from shell expansions.
By contrast, the * must be unquoted so as to trigger pathname expansion.

edited May 10, 2015 at 14:40

answered May 10, 2015 at 14:33

mklement0

453k68 gold badges729 silver badges989 bronze badges

2 Comments

Tom Fenech Over a year ago

Exactly. At least our stories check out!

mklement0 Over a year ago

@TomFenech: :) ++ for providing a better explanation.

Collectives™ on Stack Overflow

BASH: append to array

2 Answers 2

5 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related