You can use the python ThreadPoolExecutor link
Here is the general program which is not perfect but if you fill this should work
# import or some variable from your code mpimg
def read_img(name):
try:
img = mpimg.imread(name)
return img
except:
return("Did not find image")
from concurrent.futures import ThreadPoolExecutor, as_completed
# suppose the files contains th 16k file names
files = ['f1.jpg', 'f2.jpg']
future_to_file = {}
images_read = []
with ThreadPoolExecutor(max_workers=4) as executor:
for file in files:
future = executor.submit(read_img, file)
future_to_file[future] = file
for future in as_completed(future_to_file):
file = future_to_file[future]
img_read = future.result()
if img_read != 'Did not find image':
images_read.append((file, img_read))
Answer from Ganesh Sutar on Stack OverflowPhilip's answer is good, but will only create a couple of processes (one reading, one computing) which will hardly max out a modern >2 core system. Here's an alternative using multiprocessing.Pool (specifically, its map method) which creates processes which do both the reading and compute aspects, but which should make better use of all the cores you have available (assuming there are more files than cores).
#!/usr/bin/env python
import multiprocessing
import scipy
import scipy.misc
import scipy.ndimage
class Processor:
def __init__(self,threshold):
self._threshold=threshold
def __call__(self,filename):
im = scipy.misc.imread(filename)
label,n = scipy.ndimage.label(im > self._threshold)
return n
def main():
scipy.misc.imsave("lena.png", scipy.misc.lena())
files = ['lena.png'] * 100
proc=Processor(128)
pool=multiprocessing.Pool()
results=pool.map(proc,files)
print results
if __name__ == "__main__":
main()
If I increase the number of images to 500, and use the processes=N argument to Pool, then I get
Processes Runtime
1 6.2s
2 3.2s
4 1.8s
8 1.5s
on my quad-core hyperthreaded i7.
If you got into more realistic use-cases (ie actual different images), your processes might be spending more time waiting on the image data to load from storage (in my testing, they load virtually instantaneously from cached disk) and then it might be worth explicitly creating more processes than cores to get some more overlap of compute and load. Only your own scalability testing on a realistic load and HW can tell you what's actually best for you though.
The multiprocessing package is pretty easy to use. Look at the Queues example for a guide. You'll be following the producer consumer model. You want one (or more) producer processes reading images, and one (or more) consumer processes doing the image processing.
Your example would look something like this:
from multiprocessing import Process, Queue
import scipy
def process_images(q):
while not q.empty():
im = q.get()
# Do stuff with item from queue
def read_images(q, files):
for f in files:
q.put(scipy.misc.imread(f))
if __name__ == '__main__':
q = Queue()
producer = Process(target=read_images, args=(q, files))
producer.start()
consumer = Process(target=process_images, args=(q, ))
consumer.start()
This is a bit simpler than your original idea. In this example the producer adds to the queue as fast as it can rather than just staying one ahead of the consumer. That might be a problem if the producer gets so far ahead that you don't have enough memory to hold the queue. If problems arise you can get deeper into the multiprocessing docs, but this should be enough to get you started.
I did a Python version as well, it's not quite as fast but it is maybe closer to your heart :-)
#!/usr/bin/env python3
import cv2
import numpy as np
from multiprocessing import Pool
def doOne(params):
"""Append the two input images side-by-side to output the third."""
imA = cv2.imread(params[0], cv2.IMREAD_UNCHANGED)
imB = cv2.imread(params[1], cv2.IMREAD_UNCHANGED)
res = np.hstack((imA, imB))
cv2.imwrite(params[2], res)
if __name__ == '__main__':
# Build the list of jobs - each entry is a tuple with 2 input filenames and an output filename
jobList = []
for i in range(1000):
# Horizontally append a-XXXXX.png to b-XXXXX.png to make c-XXXXX.png
jobList.append( (f'a-{i:05d}.png', f'b-{i:05d}.png', f'c-{i:05d}.png') )
# Make a pool of processes - 1 per CPU core
with Pool() as pool:
# Map the list of jobs to the pool of processes
pool.map(doOne, jobList)
You can do this a little quicker with libvips. To join two images left-right, enter:
vips join left.png out.png result.png horizontal
To test, I made 200 pairs of 1200x800 PNGs like this:
for i in {1..200}; do cp x.png left$i.png; cp x.png right$i.png; done
Then tried a benchmark:
time parallel vips join left{}.png right{}.png result{}.png horizontal ::: {1..200}
real 0m42.662s
user 2m35.983s
sys 0m6.446s
With imagemagick on the same laptop I see:
time parallel convert left{}.png right{}.png +append result{}.png ::: {1..200}
real 0m55.088s
user 3m24.556s
sys 0m6.400s