0

I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total 96 cores) despite only running a few jobs at a time. The most jobs I am able to run at a time without slowing things down and increasing iowait is 2. I feel like it may have something to do with the fact that it is writing so many files, but I haven't found a solution.

#!/bin/bash 

task() {
x=$1
SUB_stats="path/to/subject/data/"$x"_cond3.feat/stats/"

cd $SUB_stats

3dcalc -prefix precentral_top_N_voxels_b -a Precentral_b+orig -b T_top_N_voxels_precentral+orig -expr 'a*b'

3dcalc -prefix occipital_top_N_voxels_b -a Occipital_b+orig -b T_top_N_voxels_occipital+orig -expr 'a*b'

3dcalc -prefix temporal_top_N_voxels_b -a Temporal_b+orig -b T_top_N_voxels_temporal+orig -expr 'a*b'


3dmaskave -quiet -sigma precentral_top_N_voxels_b+orig[0..390] > avgroi_precentral_subbrik_topvoxels.txt

3dmaskave -quiet -sigma occipital_top_N_voxels_b+orig[0..390] > avgroi_occipital_subbrik_topvoxels.txt

3dmaskave -quiet -sigma temporal_top_N_voxels_b+orig[0..390] > avgroi_temporal_subbrik_topvoxels.txt

} 

export -f task
cat young_fsf.txt | parallel --nice 15 --progress --eta -j 2 task
Improve this question
2
  • It seems to me like you've led us to proposed answers, none of which are shell scripts. If the processes are I/O intensive and you know that the processes write lots of files, what do you imagine a solution to be? Force the processes to write to /dev/null? Buy faster disk? Commented May 27, 2022 at 18:38
  • Sorry, I am still pretty novice and this is my first time asking a question here. I was wondering if the way I am writing the script is inefficient. For example, should I be splitting up each of these lines into separate scripts for example? Is there something better for I/O intensive processes than this function format? Commented May 27, 2022 at 20:07

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.