Skip to main content
Unix & Linux

Questions tagged [gnu-parallel]

Ask Question

GNU parallel is a command line utility to run programs in parallel

273 questions
Newest Active Bountied Unanswered
Filter by
Sorted by
Tagged with
1 vote
1 answer
117 views

While experimenting with GNU parallel I found that the following cases all hang with decreasing CPU usage on a Fedora 41 VM with 8GB RAM. Is this expected behaviour? parallel --halt now,fail=1 --...
Somniar's user avatar
  • 113
1 vote
1 answer
476 views

I want to run a command in parallel on a bunch of files as part of a Github CI workflow (on an ubuntu runner) in order to speed up the CI job. I would also like the parallel command to report its ...
JSQuareD's user avatar
  • 113
1 vote
1 answer
201 views

I would like to use GNU Parallel to process a huge .gz or .bz2 file. I know I can do: bzcat huge.bz2 | parallel --pipe ... But it would be nice if there was a way similar to --pipe-part that can read ...
Ole Tange's user avatar
  • 37.6k
2 votes
1 answer
63 views

I'm trying to use GNU parallel to run a command for all combinations of several behavior-changing flags: parallel 'cmd --foo {1} --bar {2} {3} out.foo={1}.bar={2}/{3/}' ::: 0 1 ::: 0 1 ::: in/* This ...
intelfx's user avatar
  • 5,759
1 vote
2 answers
108 views

I have a very slow samba share that i can access through wsl on my windows laptop. On windows accessing the share doesn't take much time but on linux it is in order of seconds. I basically replicate ...
Rouge a's user avatar
  • 11
1 vote
1 answer
86 views

I'm using GNU parallel like this (:::: is a form of --arg-file): parallel -0Xuj1 my-command -- :::: <(find … -print0) But it seems like the command's standard input is managed by GNU parallel, ...
Tobu's user avatar
  • 6,809
2 votes
2 answers
158 views

In my bash script, I need to execute two different functions, taskA and taskB, which take an integer ($i) as an argument. Since taskB $i depends on the completion of taskA $i, the following ...
Schnarco's user avatar
  • 121
1 vote
1 answer
108 views

Scenario: $ process(){ echo "[$1] [$2] [$3]" ; } ; export -f process $ process "x" "" "a.txt" [x] [] [a.txt] Here we see that the 2nd argument is empty string ...
pmor's user avatar
  • 757
2 votes
1 answer
308 views

I have a very large SQL dumpfile (30GB) that I need to edit (do some find/replace) before loading back into the database. Besides having a large size, the file also contains very long lines. Except ...
fernan's user avatar
  • 23
0 votes
1 answer
82 views

Fast and simple. This command works locate -i mymovieormysong|parallel mplayer the song (or movie) play, but i cannot control mplayer with keyboard. How to do (if possible) this? Actually when i use ...
elbarna's user avatar
  • 14.5k
1 vote
1 answer
94 views

Scenario: $ cat libs.txt lib.a lib1.a $ cat t1a.sh f1() { local lib=$1 stdbuf -o0 printf "job for $lib started\n" sleep 2 stdbuf -o0 printf "job for $...
pmor's user avatar
  • 757
2 votes
1 answer
74 views

Scenario: $ cat libs.txt lib.a lib1.a $ cat t1a.sh f1() { local lib=$1 stdbuf -o0 printf "job for $lib started\n" sleep 2 stdbuf -o0 printf "job for $...
pmor's user avatar
  • 757
4 votes
1 answer
1k views

This question is a follow-up to How to download multiple files simultaniously with wget? Similar to that question, I need to download many files. With the accepted answer above recommends the use of ...
sneumann's user avatar
  • 41
1 vote
2 answers
996 views

To extract all 7z archives into their own folder, using the archive name as the folder name, I run the command: 7z x "*.7z" -o* E.g., if one has the two files a.7z and b.7z, then folders a ...
Franck Dernoncourt's user avatar
  • 5,583
0 votes
3 answers
179 views

I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel My current code is working fine for ...
user96368's user avatar
  • 11
2 votes
1 answer
207 views

Consider the data from the GNU parallel manual's example for --group-by: cat > table.csv <<"EOF" UserID, Consumption 123, 1 123, 2 12-3, 1 221, 3 221, 1 2/21, 5 EOF ...
Iain Samuel McLean Elder's user avatar
  • 342
2 votes
2 answers
532 views

I'm trying to use GNU Parallel to run a command for each input argument, using that argument as the command's working directory (without appending it to the command line). Basically, what I need to do ...
intelfx's user avatar
  • 5,759
4 votes
2 answers
570 views

The default output mode of GNU parallel is --group: The output of each job is written to a temporary file and passed to the output of parallel only after the job has finished. When using this default ...
Juergen's user avatar
  • 764
5 votes
1 answer
1k views

Some time ago I learned about the software parallel: https://forums.servethehome.com/index.php?threads/mdadm-create-raid-0-quick-format.41161/#post-389210 Now I have almost the same goal as previous, ...
Jelle's user avatar
  • 53
0 votes
3 answers
333 views

Is it possible to run the linux split command with GNU parallel to speed up the splitting of the file? Read a compressed file, and split it into some equal parts either by line count or filesize. I'm ...
Abhishek Jain's user avatar
  • 113
1 vote
2 answers
132 views

I have a process like this that generates a predefined number of files but at random intervals: #!/bin/bash for i in {1..10} do sleep $(shuf -i 20-60 -n 1) echo $i > file_$i.txt done I have ...
Ketan's user avatar
  • 9,446
2 votes
2 answers
154 views

I want to run some script over powers of two in parallel. Doing so by giving GNU Parallel a list of the powers of two I want works well: %>parallel echo {} ::: 32, 64, 128, 256, 512, 1024 32 64 128 ...
Will's user avatar
  • 23
2 votes
1 answer
287 views

I have the following script which gets the absolute paths of some directories and pipes them into GNU parallel for zipping. I keep getting a signal 13 error and I'm not sure why. find $directory -...
testing09's user avatar
  • 33
0 votes
3 answers
206 views

I can't append to an array when I use parallel, no issues using a for loop. Parallel example: append() { arr+=("$1"); } export -f append parallel -j 0 append ::: {1..4} declare -p arr ...
Nickotine's user avatar
  • 554
1 vote
1 answer
86 views

I'm trying to wrap my head around running parallel in parallel, and it seems like I have a situation where that would be an ideal solution. I want to run a set of jobs in series - call them A-1, A-2, ...
Larry's user avatar
  • 140
2 votes
0 answers
123 views

This is an ongoing project that has undergone a major update to speed up list processing. Here is the homepage for anyone interested! It processes lists defined in lists.json by transforming the list ...
T145's user avatar
  • 223
1 vote
2 answers
120 views

To set the stage, here is an example file structure: test/1 test/2 test/3 I want to do this: find -P -O3 "test/" -type f | parallel --use-cpus-instead-of-cores -j+0 --tmpdir "test/&...
T145's user avatar
  • 223
3 votes
2 answers
771 views

I have an input file, names.txt, with the 1 word per line: apple abble aplle With my bash script I am trying to achieve the following output: apple and apple apple and abble apple and aplle ...
duda13's user avatar
  • 33
1 vote
1 answer
336 views

So, let's say I have an array with a list of URLs, and I want to use something such as GNU parallel to download the URLs in parallel. A command like this would do the trick. parallel -u wget -qc --...
Eduardo Perez's user avatar
  • 161
1 vote
1 answer
89 views

I'm using gnu parallel like below: find conf/20230214/ -name "conf.yaml" | parallel run_backtest the run_backtest is a function which will read the conf.yaml and run some code. When there ...
user560657's user avatar
  • 11
0 votes
1 answer
1k views

what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the ...
elmerjfudd's user avatar
  • 215
1 vote
1 answer
1k views

Does GNU Parallel start a batch of as many jobs as possible (the number of jobs started being governed by GNU Parallel internals or/and the -j option along with given parameters), and once complete, ...
Nikos Alexandris's user avatar
  • 1,470
1 vote
1 answer
418 views

I use GNU Parallel along a for loop like this: for BAND in $(seq 1 "$BANDS") ;do # Do not extract, unscale and merge if the scaled map exists already! SCALED_MAP="era5_and_land_${...
Nikos Alexandris's user avatar
  • 1,470
2 votes
1 answer
2k views

How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
Nikos Alexandris's user avatar
  • 1,470
2 votes
2 answers
1k views

How to compress all files in the current directory using lz4 using GNU parallel? I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?
pmor's user avatar
  • 757
1 vote
1 answer
635 views

parallel --joblog /tmp/log exit ::: 1 2 3 0 cat /tmp/log Ηow can Ι use a filter to write only failed job in my load when using GNU parallel or is there a way to get only failed jobs from the above ...
lc wanigasinghe's user avatar
  • 25
3 votes
2 answers
2k views

I'd like to just check the status of a bunch of Git repos with a quick command like parallel git -C {} status --short ::: ~/*/.git/... But the Git status doesn't include the repo name or path, so I'd ...
l0b0's user avatar
  • 53.6k
0 votes
3 answers
329 views

How safe is it to use export in bash scripts when using GNU Parallel? I have a parent script. parent.sh (echo child.sh & echo child_two.sh) || parallel bash wait if [[ "$STATUS1" == &...
lc wanigasinghe's user avatar
  • 25
2 votes
2 answers
3k views

I need to run an executable a large number of times, each time with two command line arguments. I've used to use xargs for this purpose, but lately I've been made aware of the existence of GNU ...
Tor's user avatar
  • 145
1 vote
2 answers
531 views

I have the following command in my Makefile parallel \ --eta \ --bar \ --joblog mnist/embedder.joblog \ pipenv run python3 \ -m mnist.train_embedder \ --...
nalzok's user avatar
  • 441
1 vote
2 answers
453 views

I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time. I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and ...
mankoff's user avatar
  • 295
1 vote
1 answer
237 views

When I try to write a pipeline like this: git branch | rg '^\*' | parallel git pull {} I run into a problem with whitespace. Because the branch names have leading whitespace, parallel ends up ...
Dommondke's user avatar
  • 153
3 votes
1 answer
1k views

I found a script on GitHub which I've slightly modified to fit the needs of the program I'm trying to run in a queue. It is not working however and I'm not sure why. It never actually echos the jobs ...
user avatar
0 votes
0 answers
149 views

I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total ...
mackenzie taylor's user avatar
  • 1
1 vote
2 answers
2k views

#!/usr/bin/bash TARGETS=( "81.176.235.2" "81.176.70.2" "78.41.109.7" ) myIPs=( "185.164.100.1" "185.164.100.2" "185.164.100.3" "185....
John Smith's user avatar
  • 827
1 vote
1 answer
386 views

I ran into a strange problem.I am running this example from this https://www.gnu.org/software/parallel/parset.html. But it is not working inside the script file. parset myarray seq 3 ::: 4 5 6 echo &...
Sanchu Varkey's user avatar
  • 57
1 vote
2 answers
766 views

Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5. For file: foo.txt i want to create file: foo.md5 with ...
watbywbarif's user avatar
  • 131
1 vote
1 answer
433 views

This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act". #!/bin/bash dup_chk() { # $1: f_src,...
chuck lee's user avatar
  • 11
3 votes
1 answer
302 views

I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each ...
Henrik supports the community's user avatar
  • 5,992
1 vote
1 answer
174 views

Introduction I have a bash script to execute a command in multiple servers through ssh. It uses GNU parallel in the parallel version, a for loop in the sequential one. The script is used like this: ...
IvanProsperi94's user avatar
  • 23

1
2 3 4 5 6