You can simply use multiprocessing.Pool:
from multiprocessing import Pool
def process_image(name):
sci=fits.open('{}.fits'.format(name))
<process>
if __name__ == '__main__':
pool = Pool() # Create a multiprocessing Pool
pool.map(process_image, data_inputs) # process data_inputs iterable with pool
Answer from alko on Stack OverflowYou can simply use multiprocessing.Pool:
from multiprocessing import Pool
def process_image(name):
sci=fits.open('{}.fits'.format(name))
<process>
if __name__ == '__main__':
pool = Pool() # Create a multiprocessing Pool
pool.map(process_image, data_inputs) # process data_inputs iterable with pool
You can use multiprocessing.Pool:
from multiprocessing import Pool
class Engine(object):
def __init__(self, parameters):
self.parameters = parameters
def __call__(self, filename):
sci = fits.open(filename + '.fits')
manipulated = manipulate_image(sci, self.parameters)
return manipulated
try:
pool = Pool(8) # on 8 processors
engine = Engine(my_parameters)
data_outputs = pool.map(engine, data_inputs)
finally: # To make sure processes are closed in the end, even if errors happen
pool.close()
pool.join()
parallel computing - Parallelizing a for-loop in Python - Computational Science Stack Exchange
Is there a simple way to run a loop in parallel
In Python, is it possible to use multiprocessing to speed up a for-loop like this?
parallel processing - How do I parallelize a simple Python loop? - Stack Overflow
Joblib does what you want. The basic usage pattern is:
from joblib import Parallel, delayed
def myfun(arg):
do_stuff
return result
results = Parallel(n_jobs=-1, verbose=verbosity_level, backend="threading")(
map(delayed(myfun), arg_instances))
where arg_instances is list of values for which myfun is computed in parallel.
The main restriction is that myfun must be a toplevel function.
The backend parameter can be either "threading" or "multiprocessing".
You can pass additional common parameters to the parallelized function. The body of myfun can also refer to initialized global variables, the values which will be available to the children.
Args and results can be pretty much anything with the threading backend but results need to be serializable with the multiprocessing backend.
Dask also offers similar functionality. It might be preferable if you are working with out of core data or you are trying to parallelize more complex computations.
What you're looking for is Numba, which can auto parallelize a for loop. From their documentation
from numba import jit, prange
@jit
def parallel_sum(A):
sum = 0.0
for i in prange(A.shape[0]):
sum += A[i]
return sum
If each iteration in the loop doesn't depend on the previous one, is there any way to run the loop in parallel?
I've been reading about it and it seems complex process, or can it be impementd in few lines of code?
For example:
list_ = [1,2,3,4,5,6,7,8,9,10,11,12]
def function1(element):
result = element*element
print(result)
for element in list_:
function1(element)My intended result:
1 4 9 16 25 36 49 64 81 100 121 144
Is it possible to perform multiprocessing within a for-loop, where 4 processes can run simultaneously, utilizing all 4 cores of the CPU?
The CPython implementation currently has a global interpreter lock (GIL) that prevents threads of the same interpreter from concurrently executing Python code. This means CPython threads are useful for concurrent I/O-bound workloads, but usually not for CPU-bound workloads. The naming calc_stuff() indicates that your workload is CPU-bound, so you want to use multiple processes here (which is often the better solution for CPU-bound workloads anyway, regardless of the GIL).
There are two easy ways of creating a process pool into the Python standard library. The first one is the multiprocessing module, which can be used like this:
pool = multiprocessing.Pool(4)
out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))
Note that this won't work in the interactive interpreter due to the way multiprocessing is implemented.
The second way to create a process pool is concurrent.futures.ProcessPoolExecutor:
with concurrent.futures.ProcessPoolExecutor() as pool:
out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))
This uses the multiprocessing module under the hood, so it behaves identically to the first version.
from joblib import Parallel, delayed
def process(i):
return i * i
results = Parallel(n_jobs=2)(delayed(process)(i) for i in range(10))
print(results) # prints [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
The above works beautifully on my machine (Ubuntu, package joblib was pre-installed, but can be installed via pip install joblib).
Taken from https://blog.dominodatalab.com/simple-parallelization/
Edit on Mar 31, 2021: On joblib, multiprocessing, threading and asyncio
joblibin the above code usesimport multiprocessingunder the hood (and thus multiple processes, which is typically the best way to run CPU work across cores - because of the GIL)- You can let
joblibuse multiple threads instead of multiple processes, but this (or usingimport threadingdirectly) is only beneficial if the threads spend considerable time on I/O (e.g. read/write to disk, send an HTTP request). For I/O work, the GIL does not block the execution of another thread - Since Python 3.7, as an alternative to
threading, you can parallelise work with asyncio, but the same advice applies like forimport threading(though in contrast to latter, only 1 thread will be used; on the plus side,asynciohas a lot of nice features which are helpful for async programming) - Using multiple processes incurs overhead. Think about it: Typically, each process needs to initialise/load everything you need to run your calculation. You need to check yourself if the above code snippet improves your wall time. Here is another one, for which I confirmed that
joblibproduces better results:
import time
from joblib import Parallel, delayed
def countdown(n):
while n>0:
n -= 1
return n
t = time.time()
for _ in range(20):
print(countdown(10**7), end=" ")
print(time.time() - t)
# takes ~10.5 seconds on medium sized Macbook Pro
t = time.time()
results = Parallel(n_jobs=2)(delayed(countdown)(10**7) for _ in range(20))
print(results)
print(time.time() - t)
# takes ~6.3 seconds on medium sized Macbook Pro
I usually use code of this form:
#!/usr/bin/env python3
import itertools
import multiprocessing
#Generate values for each parameter
a = range(10)
b = range(10)
c = range(10)
d = range(10)
#Generate a list of tuples where each tuple is a combination of parameters.
#The list will contain all possible combinations of parameters.
paramlist = list(itertools.product(a,b,c,d))
#A function which will process a tuple of parameters
def func(params):
a = params[0]
b = params[1]
c = params[2]
d = params[3]
return a*b*c*d
#Generate processes equal to the number of cores
pool = multiprocessing.Pool()
#Distribute the parameter sets evenly across the cores
res = pool.map(func,paramlist)
If you use a tool that makes it easy to parallelize two nested loops, but not four, you can use itertools.product to reduce four nested for loops into two:
from itertools import product
for a, b in product(a_grid, b_grid):
for c, d in product(c_grid, d_grid):
do_some_stuff(a, b, c, d)