If I have 7 methods and 2 threads. Method 1 and Method 2 need to be executed by the 2 threads(Thread 1, Thread2 ). Depending on which thread completes first.Eg: If thread 2 finishes execution first it should pick method 3 for execution and thread 1 should pick up method 4 and it should continue the same till all 7 methods are executed. Is there a way to accomplish this? Any reference would be highly appreciated
3 Answers
Yes, you can simply put the methods in a queue and let two threads compete for executing the methods:
import threading, queue
def worker(q):
while True:
method = q.get()
if not method:
return
method()
q = queue.Queue()
methods = [m1, m2, m3, m4, m5, m6, m7]
for method in methods:
q.put(method)
threads = [threading.Thread(target=worker, args=(q,)) for _ in range(2)]
for thread in threads:
q.put(None) # include a terminating signal for each thread
thread.start()
for thread in threads:
thread.join()
Comments
In addition to the queue-based solution from the other answer, you can use the excellent concurrent.futures module to do the heavy lifting for you:
with concurrent.futures.ThreadPoolExecutor(2) as pool:
for m in methods:
pool.submit(m)
The submit() method on the thread pool submits the task to the pool and returns a Future object (not to be confused with the asyncio type of the same name) that allows you to query whether a particular task is done, wait for it to finish, and pick up the result.
If you don't do any of that, the with block will automatically wait for all the submitted tasks to finish at the end of the block.
Comments
The answer submitted by @user4815162342 is certainly a simple way to go, but I thought I would show you the equivalent way with the multiprocessing.pool module:
from multiprocessing.pool import ThreadPool
with ThreadPool(2) as pool:
for m in methods:
pool.apply_async(m)
pool.close()
pool.join()
The apply_asyc method returns an AsyncResult object whose get method can be called to await the completion of the submitted "task" and to get the return value if the methods you are calling return anything of interest. Otherwise, calling pool.close() followed by pool.join() will block until all submitted tasks complete. If the methods did return values you were interested in, then you should save the AsyncResult objects returned by apply_async and call get on them instead. In that case there is no real need anymore to call close and join:
with ThreadPool(2) as pool:
results = [pool.apply_async(m) for m in methods]
return_values = [result.get() for result in results]
At the end of the with block, all the threads in the pool are terminated along with any tasks that have been submitted, i.e. pool.terminate() is implicitly called (unlike the concurrent.futures module where outstanding tasks' completions are awaited).
Clarification
There are two parallel classes, one for a process pool:
from multiprocessing import Pool
# or from multiprocessinf.pool import Pool
And one for a thread pool:
from multiprocessing.pool import ThreadPool
# or from multiprocessing.dummy import Pool
They are supposed to have the same API's (and, in fact, they do). However, pool.terminate() cannot in the case of a thread pool actually terminate threads as there is no method for killing threads. So if the thread is already running a submitted task at the time pool.terminate() is called either explicitly or implicitly, it may continue to run.