Parallel processing of variable argument list

Question

I have a makefile that is running a custom program on large numbers of input file. The program has large startup cost, and is more efficient processing multiple files in the same run

I would like to leverage make parallel processing, so i wrote a simple makefile, which I execute with make -j

In = f1.in f2.in …
Out = $(in:.in=.out)
 
All: $(out)

%.out: %.in
    Program $< > $@

I would like to speed up processing by invoking ‘program’ N times, each with multiple files.

My question: is there away to split the $(in) into N lists, each containing about the same number of files, so i can write the efficient to execute makefile. Am ok with fixed N value like 8.

All: l1.out l2.out … l8.out

l1.out: $(select 1, 8, $(in))
  Program $^ > $@

Which will work well with make -j: parralel execution, without excessive invocation of ‘program’

Needless to say: looking for solution that will NOT result in multiple invocation of external helpers like bash/awk. Ideally, a solution that will not invoke any external helpers.

Have you considered a pattern rule with multiple targets (and prerequisites)? — Andreas
– Andreas, Commented Jan 28 at 11:52

raspy · Accepted Answer · 2025-04-16 13:57:47Z

With some word list manipulation and recursive call, we could split the list into parts of a given length, e.g.:

$ cat Makefile
in := $(wildcard *.in)
partition_length := 2

# $(1) - source to split
# $(2) - length of each part
define split
  $(eval partno := $(words $(parts)))
  part$(partno) := $(wordlist 1,$(2),$(1))
  $(eval parts += part$(partno))
  remaining := $$(wordlist $$(words $$(part$$(partno)) next),$$(words $(1)),$(1))
  $$(if $$(remaining),$$(eval $$(call split,$$(remaining),$(2))))
endef

$(eval $(call split,$(in),$(partition_length)))

.SECONDEXPANSION:
part%.out: $$(part$$*)
        @echo Building $@ from $^

.PHONY: all
all: $(addsuffix .out,$(parts))
        @echo Parts = $(parts)

This would split the given list to variables called part0, part1 and so on, while keeping list of all such variables in parts. This may be used to define rules to do the actual work. In this example, creating 5 files would make it process in 3 targets, 2 for each:

$ touch 1.in 2.in 3.in 4.in 5.in
$ make
Building part0.out from 1.in 2.in
Building part1.out from 3.in 4.in
Building part2.out from 5.in
Parts = part0 part1 part2

Changing partition length to 3 would result in:

$ make
Building part0.out from 1.in 2.in 3.in
Building part1.out from 4.in 5.in
Parts = part0 part1

Adapt as needed.

Collectives™ on Stack Overflow

Parallel processing of variable argument list

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related