If you get the image objects from files, you can just send the filenames to the workers and let them open the images themselves.
Otherwise, you can send the image data (with Image.getdata()), along with the size and pixel format, and have the workers reconstruct the image using Image.new() and Image.putdata().
If you get the image objects from files, you can just send the filenames to the workers and let them open the images themselves.
Otherwise, you can send the image data (with Image.getdata()), along with the size and pixel format, and have the workers reconstruct the image using Image.new() and Image.putdata().
Just put the names of the files in a list, and let the worker processes handle them. The example below uses ImageMagick in a subprocess to do some image conversion from an obscure format. But the same principle can be used with PIL. Just replace the contents of the processfile() function. This is a program that I use frequently to convert DICOM files (a format used in medical imaging, from an X-ray machine in this case) to PNG format.
"""Convert DICOM files to PNG format, remove blank areas. The blank erea
removal is based on the image size of a Philips flat detector. The image
goes from 2048x2048 pixels to 1574x2048 pixels."""
import os
import sys
import subprocess
from multiprocessing import Pool, Lock
globallock = Lock()
def checkfor(args):
"""Make sure that a program necessary for using this script is
available.
Arguments:
args -- string or list of strings containing a command to test
"""
if isinstance(args, str):
args = args.split()
try:
f = open('/dev/null')
subprocess.call(args, stderr=subprocess.STDOUT, stdout=f)
f.close()
except:
print "Required program '{}' not found! exiting.".format(args[0])
sys.exit(1)
def processfile(fname):
"""Use the convert(1) program from the ImageMagick suite to convert the
image and crop it.
Arguments:
fname -- string containing the name of the file to process
"""
size = '1574x2048'
args = ['convert', fname, '-units', 'PixelsPerInch', '-density', '300',
'-crop', size+'+232+0', '-page', size+'+0+0', fname+'.png']
rv = subprocess.call(args)
globallock.acquire()
if rv != 0:
print "Error '{}' when processing file '{}'.".format(rv, fname)
else:
print "File '{}' processed.".format(fname)
globallock.release()
def main(argv):
"""Main program.
Arguments:
argv -- command line arguments
"""
if len(argv) == 1:
# If no filenames are given, print a usage message.
path, binary = os.path.split(argv[0])
print "Usage: {} [file ...]".format(binary)
sys.exit(0)
# Verify that the convert program that we need is available.
checkfor('convert')
# Apply the processfile() function to all files in parallel.
p = Pool()
p.map(processfile, argv[1:])
p.close()
if __name__ == '__main__':
main(sys.argv)
How are Python multithreading and multiprocessing related?
What's the difference between Python threading and multiprocessing?
What is a thread in Python?
You cannot speed it up using threading due to the Global Interpreter Lock. Certain internal state of the Python interpreter is protected by that lock, which prevents different threads that need to modify that state from running concurrently.
You could speed it up by spawning actual processes using multiprocessing. Each process will run in its own interpreter, thus circumventing the limitation of threads. With multiprocessing, you can either use shared memory, or give each process its own copy/partition of the data.
Depending on your task, you can either parallelize the processing of a single image by partitioning it, or you can parallelize the processing of a list of images (the latter is easily done using a pool). If you want to use the former, you might want to store the image in an Array that can be accessed as shared memory, but you'd still have to solve the problem of where to write the results (writing to shared memory can hurt performance badly). Also note that certain kinds of communication between processes (Queues, Pipes, or the parameter/return-value passing of some function in the module) require the data to be serialized using Pickle. This imposes certain limitations on the data, and might create significant performance-overhead (especially if you have many small tasks).
Another way for improving performance of such operations is to try writing them in Cython, which has its own support for parallelization using OpenMP - I have never used that though, so I don't know how much help it can be.
Take a look at Doug Hellmans tutorial on multiprocessing. As Björn points out, there are various issues regarding parallel processing which you'll need to get a hang of, but it can really be worth the effort.
Tip: You can use multiprocessing.cpu_count() to check the number of cores available to you.