I would suggest looking at Pools to easily create processes on the fly. If you need to have some shared state, in this case a boolean indicating a non-black image has been found, look at Managers.
Update: Here is an example of what I mean.
import multiprocessing.Manager as Manager
import multiprocessing.Pool as Pool
m = Manager()
p = Pool(processes=5)
state_info = m.dict()
state_info['image_found'] = False
def processImage(img):
# ... Process Image ...
if imageIsBlack(img):
state_info['image_found'] = True
p.terminate()
p.apply(processImage, imageList)
if state_info['image_found']:
print 'There was a black image!!'
else:
print 'No black images were found.'
Answer from owobeid on Stack OverflowI would suggest looking at Pools to easily create processes on the fly. If you need to have some shared state, in this case a boolean indicating a non-black image has been found, look at Managers.
Update: Here is an example of what I mean.
import multiprocessing.Manager as Manager
import multiprocessing.Pool as Pool
m = Manager()
p = Pool(processes=5)
state_info = m.dict()
state_info['image_found'] = False
def processImage(img):
# ... Process Image ...
if imageIsBlack(img):
state_info['image_found'] = True
p.terminate()
p.apply(processImage, imageList)
if state_info['image_found']:
print 'There was a black image!!'
else:
print 'No black images were found.'
Finally this works nicely for me. Copied it from an example here. For illustration purpose I have replaced my _isImgNonBlack function and the image sequence as a list of 0's and 1's, where 0 being a black image and 1 non-black image.
import multiprocessing
def isImgNonBlack(result_queue, imgSeq):
for img in imgSeq:
# If a non-black is found put a result
if img==1:
result_queue.put(1)
# else put a zero as the result
result_queue.put(0)
if __name__ == '__main__':
processs = []
result_queue = multiprocessing.Queue()
nbProc = 20
# making a fake list of images with
# 10,000 0's follwed by a single 1
images = [0 for n in range(10000)]
images.append(1)
for n in range(nbProc): # start processes crawling for the result
process = multiprocessing.Process(target=isImgNonBlack, args=[result_queue, images])
process.start()
processs.append(process)
print 'Starting Process : %s' % process
result = result_queue.get() # waits until any of the proccess have `.put()` a result
for process in processs: # then kill them all off
process.terminate()
# finally print the result
print "Seq have a non black img: %s" % result
performance - Image processing using multiprocessing in Python - Code Review Stack Exchange
python - Multiprocessing on chunks of an image - Stack Overflow
Multiprocessing and loading images
python - Using multiprocessing on Image processing - Stack Overflow
I'd do something like this, starting with dependencies:
from multiprocessing import Pool
import numpy as np
from PIL import Image
# and some for testing
from random import random
from time import sleep
first I define a function to divide an image up into "chunks", sort of as you talked about:
def chunkit(ys, xs, blocksize=64):
for y in range(0, ys, blocksize):
yt = (y, min(ys, y + blocksize))
for x in range(0, xs, blocksize):
xt = (x, min(xs, x + blocksize))
yield yt, xt
it's a lazy iterator, so this can go on for a while.
I then define my worker function:
def dumb_func(cc):
(y0,y1), (x0,x1) = cc
# convert to floats for ease of processing
chunk = image[y0:y1,x0:x1] / 255.
# random slow down for testing
# sleep(random() ** 6)
res = chunk ** 2
# convert back to bytes for efficiency
return cc, (res * 255).astype(np.uint8)
I make sure the source array stays as close to original format as possible for efficiency and send it back in the same format (this might take some fiddling, if you're dealing with other pixel formats obviously).
then I put it together:
if __name__ == '__main__':
source = Image.open('tmp.jpeg')
image = np.asarray(source)
print("loaded", image.shape, image.dtype)
with Pool() as pool:
resit = pool.imap_unordered(
dumb_func, chunkit(*image.shape[:2]))
output = np.empty_like(image)
for cc, res in resit:
(y0,y1), (x0,x1) = cc
output[y0:y1,x0:x1] = res
im = Image.fromarray(output, 'RGB')
im.save('out.jpeg')
this churns through a 15Mpixel image in a couple of seconds, with most of that spent loading/saving the image. it could probably be a lot more clever with array strides and cache friendliness, but hope that helps!
note: I think this code relies on CPython Unix style process forking semantics to make sure the image is shared between processes efficiently. not sure what would happen if you ran it on something else
I've been working on code for basically this same thing. Right now the goal is just to replace white pixels with transparent ones, but it seems to replace the entire image so there is a bug somewhere...It doesn't get an error within the multiprocessing module anymore though, so maybe it could serve as an example of how to load a Queue and then have your worker processes work on it!
from PIL import Image
from multiprocessing import Process, JoinableQueue
from threading import Thread
from time import time
def worker_function(q, new_data):
while True:
# print("Items in queue: {}".format(q.qsize()))
index, pixel = q.get()
if pixel[0] > 240 and pixel[1] > 240 and pixel[2] > 240:
out_pixel = (0, 0, 0, 0)
else:
out_pixel = pixel
new_data[index] = out_pixel
q.task_done()
if __name__ == "__main__":
start = time()
q = JoinableQueue()
my_image = Image.open('InputImage.jpg')
my_image = my_image.convert('RGBA')
datas = list(my_image.getdata())
new_data = [0] * len(datas) # make a blank array the size of our image to fill later
print('putting image into queue')
for count, item in enumerate(datas):
q.put((count, item))
print('starting workers')
worker_count = 50
processes = []
for i in range(worker_count):
p = Process(target=worker_function, args=[q, new_data])
p.daemon = True
p.start()
print('main thread waiting')
q.join()
my_image.putdata(new_data)
my_image.save('output.png', "PNG")
end = time()
print('{:.3f} seconds elapsed'.format(end - start))
I think it's important to "protect" your code inside the if __name__ == "__main__" block otherwise the spawned processes seem to run it.
update
It looks like you need to implement a Manager() (or there are probably other ways I am ignorant of as well!). I got my code to run by altering it into:
from PIL import Image
from multiprocessing import Process, JoinableQueue, Manager
from threading import Thread
from time import time
def worker_function(q, new_data):
while True:
# print("Items in queue: {}".format(q.qsize()))
index, pixel = q.get()
if pixel[0] > 240 and pixel[1] > 240 and pixel[2] > 240:
out_pixel = (0, 0, 0, 0)
else:
out_pixel = pixel
new_data[index] = out_pixel
q.task_done()
if __name__ == "__main__":
start = time()
q = JoinableQueue()
my_image = Image.open('InputImage.jpg')
my_image = my_image.convert('RGBA')
datas = list(my_image.getdata())
# new_data = [(0, 0, 0, 0)]*len(datas)
manager = Manager()
new_data = manager.list([(0, 0, 0, 0)]*len(datas))
print(new_data)
print('putting image into queue')
for count, item in enumerate(datas):
q.put((count, item))
print('starting workers')
worker_count = 50
processes = []
for i in range(worker_count):
p = Process(target=worker_function, args=[q, new_data])
p.daemon = True
p.start()
print('main thread waiting')
q.join()
print("Saving Image")
my_image.putdata(new_data)
my_image.save('output.png', "PNG")
end = time()
print('{:.3f} seconds elapsed'.format(end - start))
Although this doesn't seem like the fastest option! I'm sure there are other ways to increase speed. My code to do the same thing with Threads looks VERY similar:
from PIL import Image
from threading import Thread
from queue import Queue
import time
start = time.time()
q = Queue()
planeIm = Image.open('InputImage.jpg')
planeIm = planeIm.convert('RGBA')
datas = planeIm.getdata()
new_data = [0] * len(datas)
print('putting image into queue')
for count, item in enumerate(datas):
q.put((count, item))
def worker_function():
while True:
# print("Items in queue: {}".format(q.qsize()))
index, pixel = q.get()
if pixel[0] > 240 and pixel[1] > 240 and pixel[2] > 240:
out_pixel = (0, 0, 0, 0)
else:
out_pixel = pixel
new_data[index] = out_pixel
q.task_done()
print('starting workers')
worker_count = 100
for i in range(worker_count):
t = Thread(target=worker_function)
t.daemon = True
t.start()
print('main thread waiting')
q.join()
print('Queue has been joined')
planeIm.putdata(new_data)
planeIm.save('output.png', "PNG")
end = time.time()
elapsed = end - start
print('{:3.3} seconds elapsed'.format(elapsed))
Yet, processing my image takes ~23 seconds with threads and ~170 seconds with multiprocessing!! I suspect this would come from the larger overhead needed to start Process objects, and the fact that my algorithm for processing each pixel is simple for now (just the if pixel[0] > 240 and pixel[1] > 240 and pixel[2] > 240: bit), so I'm likely not yielding the speed improvements that a complex pixel processing algorithm would get me. Also to note multiprocessing documentation
a single manager can be shared by processes on different computers over a network. They are, however, slower than using shared memory.
Which leads me to believe that there are alternatives that are faster.
So I am working on a script that is reading from a camera with opencv and I have multiple processes. One process is reading the camera and I have been able to share frame between them using a pipe but I need to do it faster. I found this posting but some of the answers are missing:
https://www.reddit.com/r/learnpython/comments/4u0xh8/fastest_way_to_share_numpy_arrays_between/
Can someone give me an example of what this means?
You would simply use ctypes to find this structure underlying a PhotoImage object, update its pixelPtr to point to the numpy array and adjust the pitch, pixelSize and offset to match your format of color data.