You can use the python ThreadPoolExecutor link

Here is the general program which is not perfect but if you fill this should work

# import or some variable from your code mpimg

def read_img(name):
    try:
        img = mpimg.imread(name)
        return img
    except: 
        return("Did not find image")

from concurrent.futures import ThreadPoolExecutor, as_completed

# suppose the files contains th 16k file names
files = ['f1.jpg', 'f2.jpg']
future_to_file = {}
images_read = []

with ThreadPoolExecutor(max_workers=4) as executor:
    for file in files:
        future = executor.submit(read_img, file)
        future_to_file[future] = file
    
    for future in as_completed(future_to_file):
        file = future_to_file[future]
        img_read = future.result()
        if img_read != 'Did not find image':
            images_read.append((file, img_read))

Answer from Ganesh Sutar on Stack Overflow
Top answer
1 of 2
15

Philip's answer is good, but will only create a couple of processes (one reading, one computing) which will hardly max out a modern >2 core system. Here's an alternative using multiprocessing.Pool (specifically, its map method) which creates processes which do both the reading and compute aspects, but which should make better use of all the cores you have available (assuming there are more files than cores).

#!/usr/bin/env python

import multiprocessing
import scipy
import scipy.misc
import scipy.ndimage

class Processor:
    def __init__(self,threshold):
        self._threshold=threshold

    def __call__(self,filename):
        im = scipy.misc.imread(filename)
        label,n = scipy.ndimage.label(im > self._threshold)
        return n

def main():
    scipy.misc.imsave("lena.png", scipy.misc.lena())
    files = ['lena.png'] * 100

    proc=Processor(128)
    pool=multiprocessing.Pool()
    results=pool.map(proc,files)

    print results

if __name__ == "__main__":
    main()

If I increase the number of images to 500, and use the processes=N argument to Pool, then I get

Processes   Runtime
   1         6.2s
   2         3.2s
   4         1.8s
   8         1.5s

on my quad-core hyperthreaded i7.

If you got into more realistic use-cases (ie actual different images), your processes might be spending more time waiting on the image data to load from storage (in my testing, they load virtually instantaneously from cached disk) and then it might be worth explicitly creating more processes than cores to get some more overlap of compute and load. Only your own scalability testing on a realistic load and HW can tell you what's actually best for you though.

2 of 2
14

The multiprocessing package is pretty easy to use. Look at the Queues example for a guide. You'll be following the producer consumer model. You want one (or more) producer processes reading images, and one (or more) consumer processes doing the image processing.

Your example would look something like this:

from multiprocessing import Process, Queue
import scipy

def process_images(q):
    while not q.empty():
        im = q.get()
        # Do stuff with item from queue

def read_images(q, files):
    for f in files:
        q.put(scipy.misc.imread(f))

if __name__ == '__main__':
    q = Queue()

    producer = Process(target=read_images, args=(q, files))
    producer.start()
    consumer = Process(target=process_images, args=(q, ))
    consumer.start()

This is a bit simpler than your original idea. In this example the producer adds to the queue as fast as it can rather than just staying one ahead of the consumer. That might be a problem if the producer gets so far ahead that you don't have enough memory to hold the queue. If problems arise you can get deeper into the multiprocessing docs, but this should be enough to get you started.

🌐
GitHub
gist.github.com › kastnerkyle › b20625dafc381551c405
Threaded image reader in Python · GitHub
Threaded image reader in Python · Raw · threaded_image_reader.py · This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
🌐
Kaggle
kaggle.com › code › inoryy › fast-image-pre-process-in-parallel
Fast image pre-process in parallel
April 15, 2016 - Explore and run AI code with Kaggle Notebooks | Using data from State Farm Distracted Driver Detection
🌐
Stack Overflow
stackoverflow.com › questions › 34360306 › parallel-image-processing-in-python
parallel image processing in python - Stack Overflow
December 18, 2015 - def process_image(img_names): for img_name in img_names: cv2.imread('image/'+img_name) im = read_img(img_name) tfs_im = some_function(im) # use opencv, skimage and math cv2.imwrite('new_img/'+img_name,tfs_im) if __name__ == '__main__': ### Set Working Dir wd_path = os.path.dirname(os.path.realpath(__file__)) os.chdir(wd_path+'/..') q = Queue() img_list = os.listdir('image') # split work into 8 processes processes = 8 def splitlist(inlist, chunksize): return [inlist[x:x+chunksize] for x in xrange(0, len(inlist), chunksize)] list_splitted = splitlist(img_list, len(img_list)/processes+1) for imgs
🌐
Scott H. Hawley
drscotthawley.github.io › devblog3 › 2018 › 12 › 16 › Parallelizing-Python-Simplified.html
Parallelizing Python, Simplified | Scott H. Hawley (alt. blog via fastpages)
December 16, 2018 - Shape of first image ... cv2.imread(filename) print("Finished.") For the parallel version, we’re going to have to use a global variable....
Top answer
1 of 1
1

So the way that Python's multiprocessing works (for the most part) is that you have to designate the individual threads that you want to run. I made a brief introductory tutorial here: http://will-farmer.com/parallel-python.html

In your case, what I would recommend is split tris into a bunch of different parts, each equally sized, each that represents a "worker". You can split this list with numpy.split() (documentation here: http://docs.scipy.org/doc/numpy/reference/generated/numpy.split.html).

Then for each list in tri, we use the Threading and Queue modules to designate 8 workers.

import numpy as np
# split into 8 different lists
tri_lists = np.split(tris, 8)
# Queues are threadsafe
return_values = queue.Queue()
threads = []
def color_image(q, tris, hipo, tridex):
    """ This is the function we're parallelizing """
    for tri in tris:
        return_values.put(np.mean(hipo[tridex==tri,:], axis=0))
# Now we run the jobs
for i in range(8):
    threads.append(threading.Thread(
        target=color_image,
        args=(return_values, tri_lists[i], hipo, tridex)))
# Now we have to cleanup our results
# First get items from queue
results = [item for item in return_values.queue]
# Now set values in lopo
for i in range(len(results)):
    for t in tri_lists[i]:
        lopo[tridex==t, :] = results[i]

This isn't the cleanest way to do it, and I'm not sure if it works since I can't test it, but this is a decent way to do it. The parallelized part is now np.mean(), while setting the values is not parallelized.

If you want to also parallelize the setting of the values, you'll have to have a shared variable, either using the Queue, or with a global variable.

See this post for a shared global variable: Python Global Variable with thread

Find elsewhere
🌐
PyImageSearch
pyimagesearch.com › home › blog › multiprocessing with opencv and python
Multiprocessing with OpenCV and Python - PyImageSearch
December 29, 2022 - Using multiprocessing, Pool, and map to call the process_images function on each core of the processor. Grab the results from each independent process and combine them. If you need to review Python’s multiprocessing module, be sure to refer ...
🌐
Medium
medium.com › featurepreneur › speeding-up-image-processing-using-multiprocessing-in-python-ceeced8026bf
Speeding up image processing using multiprocessing in Python | by Santhosh Kannan | featurepreneur | Medium
February 3, 2023 - However, image preprocessing is a computationally expensive task. But it can be sped up with various techniques. One of the techniques is to process the images concurrently than processing them one by one. Let us look at how to process the images concurrently using multi-threading and multiprocessing using python.
🌐
Kaggle
kaggle.com › code › rsmits › using-multiprocessing-for-image-preprocessing
Using multiprocessing for image preprocessing
Checking your browser before accessing www.kaggle.com · Click here if you are not automatically redirected after 5 seconds
🌐
Towards Data Science
towardsdatascience.com › home › latest › parallelize your python code to save time on data processing
Parallelize your python code to save time on data processing | Towards Data Science
January 28, 2025 - Now doing this process on each image is independent of each other, i.e., processing one image would not affect any other image in the folder. Hence multiprocessing could help us reduce the total time. Our total time will be reduced by a factor equal to the number of processors we use in parallel.
🌐
The Neural Base
theneuralbase.com › home › opencv › advanced course › parallel image processing
Parallel image processing | Opencv Advanced Course | The Neural Base
Skip if: Do not parallelize if: ... multiple image copies in RAM. Parallel image processing means processing multiple images at the same time across different CPU cores using Python's multiprocessing module....
🌐
O'Reilly
oreilly.com › library › view › hands-on-image-processing › 9781789343731 › a492a83d-57ba-4098-a24a-c70bf1f4b058.xhtml
Reading and displaying multiple images at once - Hands-On Image Processing with Python [Book]
November 30, 2018 - We can use the imread_collection() function of the scikit-image io module to load in a folder all images that have a particular pattern in the filename and display them simultaneously with the imshow_collection() function.
Author: Sandipan Dey
Published: 2018
Pages: 492
🌐
Toptal
toptal.com › python › beginners-guide-to-concurrency-and-parallelism-in-python
Python Multithreading Tutorial: Concurrency and Parallelism | Toptal®
May 26, 2026 - Now, to allow our method to work properly with the async programming paradigm, we’ve introduced a while loop that reads chunks of the image at a time and suspends execution while waiting for the I/O to complete. This allows the event loop to loop through downloading the different images as each one has new data available during the download. While the zen of Python tells us there should be one obvious way to do something, there are many ways in Python to introduce concurrency into our programs.
🌐
Medium
medium.com › @ageitgey › quick-tip-speed-up-your-python-data-processing-scripts-with-process-pools-cf275350163a
Quick Tip: Speed up your Python data processing scripts with Process Pools | by Adam Geitgey | Medium
June 30, 2017 - Thanks to Python’s concurrent.futures module, it only takes 3 lines of code to turn a normal program into one that can process data in parallel. Let’s say we have a folder full of photos and we want to create thumbnails of each photo. Here’s a short program that uses Python’s built-in glob function to get a list of all the jpeg files in a folder and then uses the Pillow image processing library to save out 128-pixel thumbnails of each photo:
🌐
Quora
quora.com › How-do-I-process-several-images-parallel-using-OpenCV
How to process several images parallel using OpenCV - Quora
Answer (1 of 3): This sounds like a problem where CUDA could be used nicely. As a very simple example, suppose that you have 25 20x20 images and you want to tile them together in a 5x5 grid, thus making a big 100x100 image. Assuming you have CUDA and OpenCV installed and you know how to link your...
🌐
GitHub
gist.github.com › humbhenri › 8210347
Download images in parallel using python · GitHub
January 1, 2014 - Download images in parallel using python · Raw · get-images.py · This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
🌐
GitHub
github.com › python-pillow › Pillow › issues › 2995
Multiprocessing and loading images · Issue #2995 · python-pillow/Pillow
February 5, 2018 - So, I wanted to load different images using Pool from multiprocessing. The code is: from PIL import Image from multiprocessing import Pool def f(path): return Image.open(path) impath = ['/home/usr/Desktop/img1.jp2', '/home/usr/Desktop/im...