Run large python script in parallel/multiprocessing

Question

I have a large python script (an economic model with rows > 1500) which I want to excecute in parallel on several cpu cores. All the examples for multiprocessing I found so far were about simple functions, but not whole scripts. Could you please give me a hint how to achieve this? Thanks!

Clarification: the model generates as an output a dataset for a multitude of variables. Each result is randomly different from the other model runs. Therefore I have to run the model often enough till some deviation measure is achieved (let's say 50 times). Model input is allways the same, but not the output.

Edit, got it:

import os                                                                       
from multiprocessing import Pool                                                

n_cores = 4
n_iterations = 5                                 

def run_process(process):                                                             
    os.system('python myscript.py')                                       

if __name__ == '__main__':
    p = Pool(n_cores)
    p.map(run_process, range(n_iterations))

turn your script into a function. use it same as simple scripts. — Paritosh Singh
– Paritosh Singh, Commented May 2, 2019 at 6:49
@ParitoshSingh the hand-waving is strong :P OP: you can't just throw multiple CPUs at a script; if it was that easy then people wouldn't be worrying about the GIL and we'd all be doing it. multiprocessing is a bit of a fudge, each process gets its own namespace and works independently - you then have overhead if you want them to communicate. You can always do the kind of multiprocessing you're asking about by just launching the process twice, it just ends up as duplication of effort. — roganjosh
– roganjosh, Commented May 2, 2019 at 7:18
yep, i wont deny it :P But i think that handwaving gets people in the right spot to ask more specific questions, and sometimes thats all one needs really. — Paritosh Singh
– Paritosh Singh, Commented May 2, 2019 at 7:22

Patol75 · Accepted Answer · 2019-05-02 12:04:05Z

1

If you want to use a pool of workers, I usually do the following.

import multiprocessing as mp


def MyFunctionInParallel(foo, bar, queue):
    res = foo + bar
    queue.put({res: res})
    return


if __name__ == '__main__':
    data = []
    info = {}
    num = 
    ManQueue = mp.Manager().Queue()
    with mp.Pool(processes=numProcs) as pool:
        pool.starmap(MyFunctionInParallel, [(data[v], info, ManQueue)
                                   for v in range(num)])
resultdict = {}
for i in range(num):
    resultdict.update(ManQueue.get())

To be clearer, your script becomes the body of MyFunctionInParallel. This means that you need to slightly change your script so that the variables which depend on your input (i.e. each of your models) can be passed as arguments to MyFunctionInParallel. Then, depending on what you want to do with the results you get for each run, you can either use a Queue as sketched above or for example, write your results in a file. If you use a Queue, it means that you want to be able to retrieve your data at the end of the parallel execution (i.e. in the same script execution), and I would advise to use dictionaries as a way to store your results in the Queue, as they are very flexible on the data they can contain. On the other hand, writing up your results in a file is I guess better if you wish to share them with other users/applications. You have to be careful with concurrent writing from all the workers, so as to produce a meaningful output, but writing one file per model can also be OK.

For the main part of the code, num would be the number of models you will be running, data and info some parameters which are specific (or not) to each model and numProcs the number of processes that you wish to launch. For the call to starmap, it will basically map the arguments in the list comprehension to each call of MyFunctionInParallel, allowing each execution to have different input arguments.

edited May 2, 2019 at 12:04

answered May 2, 2019 at 7:21

Patol75

4,5771 gold badge20 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

13 Comments

roganjosh Over a year ago

This is multiprocessing on a function, which is sensible. The OP is asking about entire scripts

Patol75 Over a year ago

An entire script can be modified so as to fit in the body of a function, this is actually what I have done for the multiprocessing example I am sketching here. If this does not suit the needs of the OP, then it means a pool of worker is inappropriate and more advanced multiprocessing should be used, such as decomposing the script in smaller bits for which some can be executed in parallel.

roganjosh Over a year ago

Where have you done that in this answer? In the case you talk about, I'm curious whether you actually found any speed up? How did you manage the IPC?

Patol75 Over a year ago

If you need to run the same code numerous times, then the sketch I gave above will suit your needs.

Patol75 Over a year ago

@reportgunner No worries, just misunderstanding. Answer edited. :)

|

Collectives™ on Stack Overflow

Run large python script in parallel/multiprocessing

1 Answer 1

13 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

13 Comments

Your Answer

Sign up or log in

Post as a guest

Related