Return variable from python multi-process

Question

My original code looks similar to this toy example. It creates two random matrices, multiplies them and saves the output in a list.

import numpy as np

n = 100
N = 10
out = [1] * N

for i in range(0, N):

    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    out[i] = A * B

Now I am trying to parallelize this loop using multiprocessing. I define the function g and run_complex_operations in defs.py

import numpy as np
    
def g(n):
    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    C = A * B


def run_complex_operations(operation, input, pool):
    pool.map(operation, input)
    print(input)

and parallelize the code as follows:

import defs   
import numpy as np
import multiprocessing as mp

n = 100
N = 10
l = [n] * N

if __name__ == '__main__':
    processes_pool = mp.Pool(N)
    defs.run_complex_operations(defs.g, range(N), processes_pool)

My question is: How can I save the resulting matrix C in each iteration in the list out, so that I end up with a list of 10 matrices?

Update:

Solved.

Ahmed AEK · Accepted Answer · 2022-12-09 18:22:16Z

3

the keyword return from python functions tutorial in python returns its parameter back to the caller, you can use it to return the result of the function back to out

import numpy as np
import multiprocessing as mp

def g(n):
    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    return A * B

def run_complex_operations(operation, input, pool):
    result = pool.map(operation, input)
    print(input)
    return result

n = 100
N = 10
l = [n] * N

if __name__ == '__main__':
    processes_pool = mp.Pool(N)
    out = run_complex_operations(g, range(N), processes_pool)
    print(len(out))  # 10

Edit: a small trick, if you instead want to fill l variable with the output your can use inplace assignment of lists as follows.

l[:] = run_complex_operations(g, range(N), processes_pool)

which will take the returns of the function and put each element returned to an index in the list, .... but predefining l has no real benefit over l = run_complex_operations...

small note, you are doing an element-wise multiplication, not matrix multiplication, you should use A @ B for matrix multiplication.

edited Dec 9, 2022 at 18:22

answered Dec 9, 2022 at 18:08

Ahmed AEK

23.2k3 gold badges19 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

eigenvector Over a year ago

Thank you for your answer. When I run your code, I get an error message. Please see the update in my question for the exact code I run and the error message.

Ahmed AEK Over a year ago

@eigenvector all the functions should be modified to return the result, as in my answer, the last line must contain return and then what you want to return after it in every function.

eigenvector Over a year ago

I overlooked that. Now it's working. Thank you!

Collectives™ on Stack Overflow

Return variable from python multi-process

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related