Running shell script loop in parallel

Question

I wrote an shell script which

get list of all image files from directory
create new folder if needed for new image
optimize image in order to save storage resources

I've tried to use parallel -j "$(nproc)" before mogrify but found that it was wrong, because before mogrify is used DIR and mkdir, i need instead something like & at end of mogrify but to do it only for n processes.

the current code look like:

#!/bin/bash

find $1 -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" -type f | while read IMAGE
do
    DIR="$2"/`dirname $IMAGE`
    echo "$IMAGE > $DIR"
    mkdir -p $DIR
    mogrify -path "$DIR" -resize "6000000@>" -filter Triangle -define filter:support=2 -unsharp 0.25x0.08+8.3+0.045 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB "$IMAGE"
done

exit 0

Can someone suggest what will be the right way to run such script in parallel? as each run take about 15 seconds.

Check out GNU parallel. (Site briefly offline at the moment.) — Thomas
– Thomas, Commented Jul 1, 2019 at 16:51

that other guy · Accepted Answer · 2019-07-01 16:55:25Z

2

When you have a shell loop that does some setup and invokes an expensive command, the way to parallelize it is to use sem from GNU parallel:

for i in {1..10}
do
  echo "Doing some stuff"
  sem -j +0 sleep 2
done
sem --wait

This allows the loop to run and do its thing as normal, while also scheduling the commands to run in parallel (-j +0 runs one job per CPU core).

answered Jul 1, 2019 at 16:55

that other guy

125k12 gold badges188 silver badges214 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ole Tange · Accepted Answer · 2019-07-01 19:33:06Z

Make a bash function that deals correctly with one file and call that in parallel:

#!/bin/bash

doit() {
  IMAGE="$1"
  DIR="$2"/`dirname $IMAGE`
  echo "$IMAGE > $DIR"
  mkdir -p $DIR
  mogrify -path "$DIR" -resize "6000000@>" -filter Triangle -define filter:support=2 -unsharp 0.25x0.08+8.3+0.045 -dither None -posterize 136 -quality 82 -define jpeg:fancy-upsampling=off -define png:compression-filter=5 -define png:compression-level=9 -define png:compression-strategy=1 -define png:exclude-chunk=all -interlace none -colorspace sRGB "$IMAGE"
}
export -f doit

find $1 -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" -type f |
    parallel doit

Default for GNU Parallel is to run one job per CPU-thread, so ǹproc is not needed.

This has less overhead than starting sem for each file (sem = 0.2 sec per call, parallel = 7 ms per call).

Collectives™ on Stack Overflow

Running shell script loop in parallel

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related