You can simply use multiprocessing.Pool:

from multiprocessing import Pool

def process_image(name):
    sci=fits.open('{}.fits'.format(name))
    <process>

if __name__ == '__main__':
    pool = Pool()                         # Create a multiprocessing Pool
    pool.map(process_image, data_inputs)  # process data_inputs iterable with pool
Answer from alko on Stack Overflow
๐ŸŒ
Super Fast Python
superfastpython.com โ€บ multiprocessing-pool-for-loop
Parallel For-Loop With a Multiprocessing Pool โ€“ SuperFastPython
August 15, 2022 - You can convert a for-loop to be parallel using the multiprocessing.Pool class.
Discussions

parallel computing - Parallelizing a for-loop in Python - Computational Science Stack Exchange
If you want shared memory parallelism, ... parallel loop, the multiprocessing standard library package is probably what you want, maybe with a nice front-end, like joblib, as mentioned in Doug's post. The standard library isn't going to go away, and it's maintained, so it's low-risk. There are other options out there, too, like Parallel Python and IPython's ... More on scicomp.stackexchange.com
๐ŸŒ scicomp.stackexchange.com
Is there a simple way to run a loop in parallel
Yeah. concurrent.futures is really straightforward. You just use ProcessPoolExecutor or ThreadPoolExecutor, depending on whether the thing you're doing is CPU-bound or IO-bound. It looks something like: from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor def process_task(x): return x * x # Example task tasks = [*range(20)] # Runs across multiple processes with ProcessPoolExecutor() as executor: results = list(executor.map(process_task, tasks)) # Runs across multiple threads with ThreadPoolExecutor() as executor: results = list(executor.map(process_task, tasks)) print(results) More on reddit.com
๐ŸŒ r/learnpython
25
2
March 6, 2025
In Python, is it possible to use multiprocessing to speed up a for-loop like this?
It's possible to partition the list in n parts and have n processes loop over each part, then merge the results. And while this sounds a bit tedious the map function of the multiprocessing module already implements this so it's literally two lines of code for creating a pool and calling map. The first example shows how it's done: https://docs.python.org/3/library/multiprocessing.html More on reddit.com
๐ŸŒ r/learnprogramming
1
0
March 29, 2023
parallel processing - How do I parallelize a simple Python loop? - Stack Overflow
To parallelize a simple for loop, joblib brings a lot of value to raw use of multiprocessing. More on stackoverflow.com
๐ŸŒ stackoverflow.com
๐ŸŒ
Super Fast Python
superfastpython.com โ€บ multiprocessing-for-loop
Multiprocessing For-Loop in Python โ€“ SuperFastPython
August 28, 2022 - We can use the multiprocessing.Process to create a parallel-for loop. The multiprocessing.Process class allows us to create and manage a new child process in Python.
๐ŸŒ
Scaler
scaler.com โ€บ home โ€บ topics โ€บ parallel for loop in python
Parallel for Loop in Python - Scaler Topics
May 8, 2024 - Python parallel for loop is important as they ... multiprocessing. Pool().map() is a good choice for parallelizing simple loops. To parallelize the loop the multiprocessing package provides a process pool with helpful functions to automatically manage a pool of worker processes.
๐ŸŒ
Tutor Python
tutorpython.com โ€บ 2-easy-ways-to-use-parallel-for-loop-in-python
2 Easy ways to use parallel For loop in Python - Tutor Python
July 28, 2024 - The code snippet provided demonstrates how to use the multiprocessing module to run parallel for loop. The function square(n) is a simple function that calculates the square of a number and simulates a time-consuming task by including a one-second sleep. from multiprocessing import Pool import time def square(n): time.sleep(1) # Simulate a time-consuming task return n * n
๐ŸŒ
Medium
medium.com โ€บ mlearning-ai โ€บ how-do-i-make-my-for-loop-faster-multiprocessing-multithreading-in-python-8f7c3de36801
How do I make my for loop faster? - Multiprocessing & Multithreading in Python | by Apurva Misra | Medium
May 4, 2022 - This would lead to parallel processing and a faster response, but the same amount of CPU time. Simply put, if we had 2 items on the list and it was taking 20 seconds earlier, thanks to multiprocessing it will take 10 seconds (approx) but the same amount of CPU cycles which gets divided among the two processes.
Find elsewhere
๐ŸŒ
Python
docs.python.org โ€บ 3 โ€บ library โ€บ multiprocessing.html
multiprocessing โ€” Process-based parallelism
Compared to using the Pool interface ... from waiting for the results. In multiprocessing, processes are spawned by creating a Process object and then calling its start() method....
๐ŸŒ
DEV Community
dev.to โ€บ terrillo โ€บ a-quick-introduction-to-python-multiprocessing-pools-speeding-up-your-workflow-17op
A quick introduction to Python Multiprocessing Pools. Speeding up your workflow - DEV Community
March 23, 2020 - As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced the multiprocessing module to let us write parallel code. There are plenty of classes in Python multiprocessing module for building a parallel program.
๐ŸŒ
CodeRivers
coderivers.org โ€บ blog โ€บ parallel-for-loop-python
Parallel For Loop in Python: A Comprehensive Guide - CodeRivers
August 8, 2026 - Parallel for loops offer a solution by allowing multiple iterations of a loop to run simultaneously, potentially reducing the overall execution time significantly. Python provides several approaches to parallelism, including the standard library modules threading, multiprocessing, and ...
Top answer
1 of 15
314

The CPython implementation currently has a global interpreter lock (GIL) that prevents threads of the same interpreter from concurrently executing Python code. This means CPython threads are useful for concurrent I/O-bound workloads, but usually not for CPU-bound workloads. The naming calc_stuff() indicates that your workload is CPU-bound, so you want to use multiple processes here (which is often the better solution for CPU-bound workloads anyway, regardless of the GIL).

There are two easy ways of creating a process pool into the Python standard library. The first one is the multiprocessing module, which can be used like this:

pool = multiprocessing.Pool(4)
out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))

Note that this won't work in the interactive interpreter due to the way multiprocessing is implemented.

The second way to create a process pool is concurrent.futures.ProcessPoolExecutor:

with concurrent.futures.ProcessPoolExecutor() as pool:
    out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))

This uses the multiprocessing module under the hood, so it behaves identically to the first version.

2 of 15
211
from joblib import Parallel, delayed
def process(i):
    return i * i
    
results = Parallel(n_jobs=2)(delayed(process)(i) for i in range(10))
print(results)  # prints [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The above works beautifully on my machine (Ubuntu, package joblib was pre-installed, but can be installed via pip install joblib).

Taken from https://blog.dominodatalab.com/simple-parallelization/


Edit on Mar 31, 2021: On joblib, multiprocessing, threading and asyncio

  • joblib in the above code uses import multiprocessing under the hood (and thus multiple processes, which is typically the best way to run CPU work across cores - because of the GIL)
  • You can let joblib use multiple threads instead of multiple processes, but this (or using import threading directly) is only beneficial if the threads spend considerable time on I/O (e.g. read/write to disk, send an HTTP request). For I/O work, the GIL does not block the execution of another thread
  • Since Python 3.7, as an alternative to threading, you can parallelise work with asyncio, but the same advice applies like for import threading (though in contrast to latter, only 1 thread will be used; on the plus side, asyncio has a lot of nice features which are helpful for async programming)
  • Using multiple processes incurs overhead. Think about it: Typically, each process needs to initialise/load everything you need to run your calculation. You need to check yourself if the above code snippet improves your wall time. Here is another one, for which I confirmed that joblib produces better results:
import time
from joblib import Parallel, delayed

def countdown(n):
    while n>0:
        n -= 1
    return n


t = time.time()
for _ in range(20):
    print(countdown(10**7), end=" ")
print(time.time() - t)  
# takes ~10.5 seconds on medium sized Macbook Pro


t = time.time()
results = Parallel(n_jobs=2)(delayed(countdown)(10**7) for _ in range(20))
print(results)
print(time.time() - t)
# takes ~6.3 seconds on medium sized Macbook Pro
๐ŸŒ
Super Fast Python
superfastpython.com โ€บ parallel-nested-for-loops-in-python
Parallel Nested For-Loops in Python โ€“ SuperFastPython
November 16, 2022 - The updated version of the nested-for loop that executes all tasks in parallel using a shared process pool is listed below. # SuperFastPython.com # example of a nested for-loop with a shared process pool import time import multiprocessing # third level task def task3(arg): # report message print(f'\t\t>>>task3 {arg}', flush=True) # do work time.sleep(1) # second level task def task2(arg, pool): # report message print(f'\t>>task2 {arg}', flush=True) # do work time.sleep(1) # issue third level tasks and wait pool.map(task3, range(2)) # top level task def task1(arg, pool): # report message print(
๐ŸŒ
Python
wiki.python.org โ€บ moin โ€บ ParallelProcessing
Parallel Processing and Multiprocessing in Python
Included in Python 2.6/3.0 as multiprocessing, and backported under the same name. PyCSP Communicating Sequential Processes for Python allows easy construction of processes and synchronised communication. PyMP - OpenMP inspired, fork-based framework for conveniently creating parallel for-loops and sections.
๐ŸŒ
HPC Community
hpc-community.unige.ch โ€บ hpc technical
Python for loop parallelism - HPC Technical - HPC Community
July 30, 2021 - Dear HPC community, I was wondering what is the best way to parallelize a python for loop on the Yggdrasil HPC. The iterations are independent of each other and I can not easily vectorize it. Therefore, I thought about the multiprocessing library. Now my question is what is the most efficient way to allocate resources on the cluster: increase the number of cpus per task, or the number of tasks? i was thinking about something like this: from multiprocessing import pool pool = Pool(size) jobs...
๐ŸŒ
Medium
medium.com โ€บ @mjschillawski โ€บ quick-and-easy-parallelization-in-python-32cb9027e490
Quick and Easy Parallelization in Python | by Michael Schillawski | Medium
April 27, 2018 - Parallel forks the Python interpreter into a number of processes equal to the number of jobs (and by extension, the number of cores available). Each process will run one iteration, and return the result.