Using Python multiprocessing to run object computations

Question

I need to do run some computations to update object attributes. I want to use parallel computing since I need to update multiple objects' attributes, that is, I have multiple objects and I need to do the same computation for each one. The objects do not share information between them.

I am currently using a process pool with map or a similar function, and the problem is that these processes copy the object, then do the computation, instead of just doing the computation directly using the original object. Is there any way around this?

As an example:

from multiprocessing import Pool

class A:

    def __init__(self, init):
        self.a = init

    def func(self, b):
        self.a = self.a + b

foo = A(2)
print(foo.a) # prints 2

p = Pool()
result = p.map(foo.func, (3,))
print(foo.a) #prints 2, should print 5

foo.func(3)
print(foo.a) #prints 5 as expected

Usually not. It may be possible for some special objects (like numpy arrays) which allow to use shared memory for their data. Also it is possible for primitive types using a Manager. — Michael Butscher
– Michael Butscher, Commented Jan 28, 2021 at 5:06
multiprocessing uses processes which have separated memory and it has to send data to processes (using pickle). You may try to use multiprocessing.shared_memory but I never used it. threads uses shared memory. — furas
– furas, Commented Jan 28, 2021 at 6:00
You seem to want to do some kind of reduce operation, i.e. the above example would compute a sum of all values in the given list/tuple (only (3,) in the example + the init value)... but this is not a parallel operation, i.e. you can't do N parallel additions and have a sum of N elements. — Emil Vatai
– Emil Vatai, Commented Jan 28, 2021 at 6:33
in your example you should use return self.a and display result - or return self and display result.a — furas
– furas, Commented Jan 28, 2021 at 16:48

neharty · Accepted Answer · 2021-01-28 17:54:32Z

Here's my workaround using queues:

from multiprocessing import Process, Queue
import time

class A:
    def __init__(self, init):
        self.a = init

    def func(self, b):
        self.a = self.a + b

    def par_func(self, q, b):
        print('starting')
        self.func(b)
        q.put(self.a)
        time.sleep(10)
        print('whew! all done')

foo = A(2)
bar = A(6)
print('initial:', foo.a, bar.a) # prints 2, 6

q1, q2 = Queue(), Queue()
p1, p2 = Process(target=foo.par_func, args=(q1, 3)), Process(target=bar.par_func, args=(q2, 4))
p1.start()
p2.start()
p1.join()
p2.join()

#prints from par_func() were nonsequential i.e. parallelism is occurring 

foo.a = q1.get()
bar.a = q2.get()

print('final:', foo.a, bar.a) # prints 5, 10

this is probably far from ideal, but it works for my purposes. I'm leaving it here for anyone else with this problem.

Collectives™ on Stack Overflow

Using Python multiprocessing to run object computations

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related