For-loop with multiprocessing in Python

Question

Here is the structure of the code snippet that creates the graph and meets certain conditions:

for i in range(1, len(binary_combinations)):

    # ...something          

    h=0
    while len(added)<i+1:   

         #...something  

         for j in it.combinations(good, 8):

             #...something

         h=h+1

Where binary_combinations, added and good they are some lists. I'm trying to implement multiprocessing for it for the whole for loop. Or only for the function it.combinations, but to no avail, because I can not reconcile this with the execution of the while-loop. How to approach it?

Before multiprocessing have you considered vectorizing your code? — Emmanuel-Lin
– Emmanuel-Lin, Commented Jan 5, 2018 at 9:11
I did not think about it because I'm not very good at vectorization yet. but if it would help, I will have to read more about it :) — Tomasz Przemski
– Tomasz Przemski, Commented Jan 5, 2018 at 9:14
secondly, I was interested in 'ordinary' loops because of the large number of values in the lists, so as not to remember the complete results in the operating memory. — Tomasz Przemski
– Tomasz Przemski, Commented Jan 5, 2018 at 9:17

Netwave · Accepted Answer · 2018-01-05 09:56:44Z

2

As far as can be seen, you can abstract the first for loop easily:

from multiprocessing import Pool, cpu_count
import itertools as it

def foo(i):
    global added
    global good
    h=0
    while len(added)<i+1:   
        #...something  
        for j in it.combinations(good, 8):
            #...something
            h=h+1

pool = Pool(cpu_count())
results = pool.map(foo, range())

The main problem you will have here is that if you are mutating the lists inside your #...something code blocks it would be worthless, since the processes will not be sharing their stack state.

answered Jan 5, 2018 at 9:56

Netwave

43.2k6 gold badges69 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

For-loop with multiprocessing in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related