Skip to main content
Tweeted twitter.com/StackUnix/status/970561827812921345
edited tags
Link
Jeff Schaller
  • 68.9k
  • 35
  • 122
  • 268
Source Link
david
  • 2.3k
  • 7
  • 27
  • 31

parallelize rsync calls using gnu parallel

I am trying to parallelize my rsync calls using gnu-parallel as shown below. But whenever I run my below script, it looks like it doesn't copy anything at all. But I do see lot of rsync processes (ps aux | grep rsync) running so not sure what's wrong here:

export PRIMARY=/data01/test_primary
export SECONDARY=/data02/test_secondary
export dir3=/bat/data/snapshot/20180227
PRIMARY_FILES=(685 959 682 679 688 651 909 906 657 881 884 878 853 707 847)
SECONDARY_FILES=(950 883 887 890 1001 994 997 1058 981 833)

export LOCATION_1="machineA"
export LOCATION_2="machineB"
export LOCATION_3="machineC"

do_Copy() {
  el=$1
  PRIMSEC=$2
  rsync -az golden@"$LOCATION_1":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || rsync -az golden@"$LOCATION_2":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || rsync -az golden@"$LOCATION_3":"$dir3"/proc_"$el"_5.data "$PRIMSEC"/. || exit 1
}
export -f do_Copy
parallel -j 5 do_Copy {} $PRIMARY ::: ${PRIMARY_FILES[@]} &
parallel -j 5 do_Copy {} $SECONDARY ::: ${SECONDARY_FILES[@]} &
wait

echo "All copied."

Is there anything wrong I am doing here?