I have a series of 250 bash scripts which need to be executed in parallel. Each script utilizes approximately 1 core so I do not want to execute them all at the same time. I would like to run 10 scripts at the same time and whenever one finishes execute another script.
4
-
You say bash scripts, but tag it with python. Which is it?Paul Hodges– Paul Hodges2019-04-17 16:16:49 +00:00Commented Apr 17, 2019 at 16:16
-
Sorry, I am up for any solution. I currently work with python and thought that using sub-process or multiprocessing might be a good solution but I am really unsure of the best course to pursue.magladde– magladde2019-04-17 16:23:52 +00:00Commented Apr 17, 2019 at 16:23
-
Try parallel. If you need it, I wrote a spooler script that will read a file of commands to be executed, but this is likely a better solution.Paul Hodges– Paul Hodges2019-04-17 16:30:35 +00:00Commented Apr 17, 2019 at 16:30
-
(Technically you should really submit an effort of your own... but this is a fun topic I've fought with before. Lucky you, lol)Paul Hodges– Paul Hodges2019-04-17 17:34:21 +00:00Commented Apr 17, 2019 at 17:34
Add a comment
|
1 Answer
I recommend parallel, but I'm going to post this monstrosity for the benefit of having people pick it apart and tune it. :)
#! /bin/env bash
## TODO: this documentation block needs to be expanded.
use="
$0 <#procs> <cmdfile>
Pass the number of desired processes to prespawn as the 1st argument.
Pass the command file with the list of tasks you need done.
Command file format:
KEYSTRING:cmdlist
where KEYSTRING will be used as a unique logfile name
and cmdlist is the base command string to be run
KEYSTRING may not contain whitespace of any sort.
Other lines are not allowed, including blanks or comments.
"
die() { echo "$@ $use" >&2; exit 1; }
case $# in
2) case "$1" in
*[^0-9]*) die "INVALID #procs '$1'" ;;
esac
declare -i primer="$1" # a countdown of how many processes to pre-spawn
cmdfile="$2"
[[ -r "$cmdfile" ]] || { die "$cmdfile not readable"; }
grep -v : "$cmdfile" || { die "$cmdfile has invalid lines"; }
declare -i lines=$( grep -c : $cmdfile)
if (( lines < primer ))
then die "Note - command lines in $cmdfile ($lines) fewer than requested process chains ($primer)"
fi ;;
*) die ;;
esac >&2
trap 'echo abort $0@$LINENO; use; exit 1' ERR # make sure any error is fatal
trap ': no-op to ignore' HUP # ignore hangups (built-in nohup without explicit i/o redirection)
spawn() {
IFS="$IFS:" read key cmd && [[ "${cmd:-}" ]] || return
echo "$(date) executing '$cmd'; c.f. $key.log" | tee "$key.log"
echo "# autogenerated by $0 $(date)
{ $cmd
spawn
} >> $key.log 2>&1 &
" >| $key.sh
. $key.sh
rm -f $key.sh
return 0
}
# create a command list based on those designators
declare chains=0
while (( primer-- )) # until we've filled the requested quota
do spawn # create a child process
done < $cmdfile
