multiprocessing: Run through an array with multiple processes?

Question

How to run through an array with multiple processes (for example 2) in parallel? For example I have this code:

from multiprocessing import Process, Value
import time, os

n=Value('i', 0)

l=[1, 2, 3, 4, 5]

def fun():
  try:
        while True:
            #time.sleep(0.01)   not used, explanation below
            print os.getpid(), l[n.value]
            n.value+=1
  except:
        return

for i in range(2):
    p=Process(target=fun)
    p.start()

Expected output should be like:

But I'm getting:

Only one process is running through an array.

I'm getting expected result only by putting time.sleep() with any very small value, but I'm looking for solution without it, if possible.

@information_interchange if there is no sleep - yes, only one process is running — Володимир Кельбель
– Володимир Кельбель, Commented Oct 31, 2017 at 0:42

martineau · Accepted Answer · 2017-10-31 13:09:27Z

1

The problem is because your task executes so quickly the first process executed finishes before the second one even gets started. This is why putting the call to sleep() "fixes" things—it's because it slows the task down enough to give second time enough time to start, so they can both run concurrently for a while

You can see this with your code if you make the l list a lot larger, say l = range(1, 1001).

To further illustrate this, below is a modified version of your code that also shows that eventually both of them will be running at the same time. It also prints out a little more information about what is occurring within each task:

from multiprocessing import Process, Value
import time, os

n = Value('i', 0)
l = [1, 2, 3, 4, 5]

def fun(cycles):
    assert cycles >= 0
    while cycles:
        try:
            while True:
                with n.get_lock():
                    print os.getpid(), l[n.value]
                    n.value += 1
        except IndexError:
            print('list index out of range exception occurred, resetting "n"')
            with n.get_lock():  # Reset for another cycle
                n.value = 0
        except Exception as exc:
            print('unexpected exception {} occurred'.format(exc))
            break
        cycles -= 1

    print('{} task terminating'.format(os.getpid()))


if __name__ == '__main__':
    cycles = 100
    for i in range(2):
        p = Process(target=fun, args=(cycles,))
        p.start()

    print('done')

edited Oct 31, 2017 at 13:09

answered Oct 31, 2017 at 0:50

martineau

124k29 gold badges181 silver badges319 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

information_interchange Over a year ago

This would seem to explain it, I was thinking of a lack of context switch/processor scheduling issue

Володимир Кельбель Over a year ago

Sorry I just forgot to put brackets here to in question. Btw on pc it's already calling start() and still not working as expected. Edited

martineau Over a year ago

Works for me on a Windows PC.

Володимир Кельбель Over a year ago

Hmmm... Can it be some PC issue? I'm running linux

martineau Over a year ago

I understand what you want. The results are likely because of overhead of task switching verses the fact that the target processes have an extremely tight loop running almost continuously that is causing the them to behave as they do. This would also explain why adding a small amount of sleep() to the tight loop helps.

|

Collectives™ on Stack Overflow

multiprocessing: Run through an array with multiple processes?

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related