running a function in parallel in Python

Question

I'm trying to run a function like f in parallel in Python but have two problems:

When using map, function f is not applied to all permuted tuples of arrays a and b.
When trying to use Pool, I get the following error:

TypeError: '<=' not supported between instances of 'tuple' and 'int'

def f(n,m):
    x = n * m
    return x

a = (1,2,3,4,5)
b = (3,4,7,8,9)
result = map(f, a, b)
print(list(result))

#now trying parallel computing
from multiprocessing import Pool
pool = Pool(processes=4)
print(*pool.map(f, a, b))

"function f is not applied to all permuted tuples of arrays a and b." Why do you think it should be? What does the map documentation say? Exactly which function calls do you want to be made for this input, and why? — Karl Knechtel
– Karl Knechtel, Commented Sep 2, 2021 at 2:59

Mark Tolonen · Accepted Answer · 2021-09-02 03:37:12Z

I didn't make any changes for your #1 issue and get the expected result from using map(). You seem to have an incorrect assumption of how it works, but didn't provide expectation vs. actual results for your example.

for #2 to return the same answers as #1, you need starmap() instead of map() for this instance of multiprocessing use, and then zip() the argument lists to provide sets of arguments. If on an OS that doesn't fork (and for portability if you are), run global code only if it is the main process, and not a spawned process by using the documented if __name__ == '__main__': idiom:

from multiprocessing import Pool

def f(n,m):
    x = n * m
    return x

if __name__ == '__main__':
    a = (1,2,3,4,5)
    b = (3,4,7,8,9)
    result = map(f, a, b)
    print(list(result))

    #now trying parallel computing
    pool = Pool(processes=4)
    print(*pool.starmap(f, zip(a, b)))

Output:

[3, 8, 21, 32, 45]
3 8 21 32 45

If you actually want permutations as mentioned in #1, use itertools.starmap or pool.starmap with itertools.product(a,b) as parameters instead.

Collectives™ on Stack Overflow

running a function in parallel in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related