Here's a minimal working example with optional FPS control. If you need to extract frames from your process back to the main program, you can use multiprocessing.Queue() to transfer frames since multiprocesses have an independent memory stack.
import multiprocessing as mp
import cv2, time
def capture_frames():
src = 'test.mp4'
capture = cv2.VideoCapture(src)
capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
# FPS = 1/X, X = desired FPS
FPS = 1/120
FPS_MS = int(FPS * 1000)
while True:
# Ensure camera is connected
if capture.isOpened():
(status, frame) = capture.read()
# Ensure valid frame
if status:
cv2.imshow('frame', frame)
else:
break
if cv2.waitKey(1) & 0xFF == ord('q'):
break
time.sleep(FPS)
capture.release()
cv2.destroyAllWindows()
if __name__ == '__main__':
print('Starting video stream')
capture_process = mp.Process(target=capture_frames, args=())
capture_process.start()
Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts
Python OpenCV streaming from camera - multithreading, timestamps
Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture
How to capture multiple camera streams with OpenCV?
OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?
Storing RTSP stream as video file with OpenCV VideoWriter
OpenCV video saving
Python OpenCV multiprocessing cv2.VideoCapture mp4
[Question] How does multiprocessing work in python-opencv?
opencv - Read webcam in python with multiprocessing - Stack Overflow
How many parallel VideoCapture's can one camera handle? - Python - OpenCV
VideoCapture frame is different when INSIDE and OUTSIDE a Multiprocessing Job
Hi. When I run a python VideoCapture loop, there are processes launched for each CPU core that show the python script running. How does this work under the hood in regard to the OpenCV library? Does it split the incoming video frames so they can be processed in parallel? It shows 15% resource use for each core (out of 20 cores) for a 30 FPS webcam showing simple video without processing using imshow. Can it be fit into a single process if it's only showing the frames? I couldn't find documentation on it.
I'm trying to optimize video processing without using a video card and thought the first step would be to not use all 20 CPU cores when showing video without processing, so that later the code can be used on a smaller device like a Raspberry Pi.
The simplest approach is to use the newer cv2 module that's based on NumPy arrays. That way you don't have to mess with manual pickling. Here's the fix (I just changed 4 lines of code):
import multiprocessing
import cv2
queue_from_cam = multiprocessing.Queue()
def cam_loop(queue_from_cam):
print 'initializing cam'
cap = cv2.VideoCapture(0)
print 'querying frame'
hello, img = cap.read()
print 'queueing image'
queue_from_cam.put(img)
print 'cam_loop done'
cam_process = multiprocessing.Process(target=cam_loop,args=(queue_from_cam,))
cam_process.start()
while queue_from_cam.empty():
pass
print 'getting image'
from_queue = queue_from_cam.get()
print 'saving image'
cv2.imwrite('temp.png', from_queue)
print 'image saved'
It appears that the solution was to convert the opencv iplimage object to string, then pickle it before adding it to the queue:
import multiprocessing
import cv
import Image
import pickle
import time
queue_from_cam = multiprocessing.Queue()
def cam_loop(queue_from_cam):
print 'initializing cam'
cam = cv.CaptureFromCAM(-1)
print 'querying frame'
img = cv.QueryFrame(cam)
print 'converting image'
pimg = img.tostring()
print 'pickling image'
pimg2 = pickle.dumps(pimg,-1)
print 'queueing image'
queue_from_cam.put([pimg2,cv.GetSize(img)])
print 'cam_loop done'
cam_process = multiprocessing.Process(target=cam_loop,args=(queue_from_cam,))
cam_process.start()
while queue_from_cam.empty():
pass
print 'getting pickled image'
from_queue = queue_from_cam.get()
print 'unpickling image'
pimg = pickle.loads(from_queue[0])
print 'unconverting image'
cv_im = cv.CreateImageHeader(from_queue[1], cv.IPL_DEPTH_8U, 3)
cv.SetData(cv_im, pimg)
print 'saving image'
cv.SaveImage('temp.png',cv_im)
print 'image saved'
Based on the comments on the original question I've done some testing and thought it worth sharing the (interesting) results. Big savings potential for anyone using OpenCV's VideoCapture.set(CAP_PROP_POS_MSEC) or VideoCapture.set(CAP_PROP_POS_FRAMES).
I've done some profiling comparing three options:
1. GET FRAMES BY SEEKING TO TIME:
frames = {}
def get_all_frames_by_ms(time):
while True:
video_capture.set(cv2.CAP_PROP_POS_MSEC, time)
capture_success, frames[time] = video_capture.read()
if not capture_success:
break
time += 1000
2. GET FRAMES BY SEEKING TO FRAME NUMBER:
frames = {}
def get_all_frames_by_frame(time):
while True:
# Note my test video is 12.333 FPS, and time is in milliseconds
video_capture.set(cv2.CAP_PROP_POS_FRAMES, int(time/1000*12.333))
capture_success, frames[time] = video_capture.read()
if not capture_success:
break
time += 1000
3. GET FRAMES BY GRABBING ALL, BUT RETRIEVING ONLY ONES I WANT:
def get_all_frames_in_order():
prev_time = -1
while True:
grabbed = video_capture.grab()
if grabbed:
time_s = video_capture.get(cv2.CAP_PROP_POS_MSEC) / 1000
if int(time_s) > int(prev_time):
# Only retrieve and save the first frame in each new second
self.frames[int(time_s)] = video_capture.retrieve()
prev_time = time_s
else:
break
Running through those three approaches, the timings (from three runs of each) are as follows:
- 33.78s 29.65s 29.24s
- 31.95s 29.16s 28.35s
- 11.81s 10.76s 11.73s
In each case it's saving 100 frames at 1sec intervals into a dictionary, where each frame is a 3072x1728 image, from a .mp4 video file. All on a 2015 MacBookPro with 2.9 GHz Intel Core i5 and 8GB RAM.
Conclusions so far... if you're interested in retrieving only some frames from a video, then very worth looking at running through all frames in order and grabbing them all, but only retrieving those you're interested in - as an alternative to reading (which grabs and retrieves in one go). Gave me an almost 3x speedup.
I've also re-looked at multi-threading on this basis. I've got two test processes - one that gets the frames, and another that processes them once they're available:
frames = {}
def get_all_frames_in_order():
prev_time = -1
while True:
grabbed = video_capture.grab()
if grabbed:
time_s = video_capture.get(cv2.CAP_PROP_POS_MSEC) / 1000
if int(time_s) > int(prev_time):
# Only retrieve and save the first frame in each new second
frames[int(time_s)] = video_capture.retrieve()
prev_time = time_s
else:
break
def process_all_frames_as_available(processing_time):
prev_time = 0
while True:
this_time = prev_time + 1000
if this_time in frames and prev_time in frames:
# Dummy processing loop - just sleeps for specified time
sleep(processing_time)
prev_time += self.time_increment
if prev_time + self.time_increment > video_duration:
break
else:
# If the frames aren't ready yet, wait a short time before trying again
sleep(0.02)
For this testing, I then called them either one after the other (sequentially, single threaded), or with the following muti-threaded code:
get_frames_thread = Thread(target=get_all_frames_in_order)
get_frames_thread.start()
process_frames_thread = Thread(target=process_all_frames_as_available, args=(0.02,))
process_frames_thread.start()
get_frames_thread.join()
process_frames_thread.join()
Based on that, I'm now happy that multi-threading is working effectively and saving a significant amount of time. I generated timings for the two functions above separately, and then together in both single-threaded and multi-threaded modes. The results are below (number in bracket is the time in seconds that the 'processing' for each frame takes, which in this case is just a dummy / delay):
get_all_frames_in_order - 2.99s
Process time = 0.02s per frame:
process_all_frames_as_available - 0.97s
single-threaded - 3.99s
multi-threaded - 3.28s
Process time = 0.1s per frame:
process_all_frames_as_available - 4.31s
single-threaded - 7.35s
multi-threaded - 4.46s
Process time = 0.2s per frame:
process_all_frames_as_available - 8.52s
single-threaded - 11.58s
multi-threaded - 8.62s
As you can hopefully see, the multi-threading results are very good. Essentially, it takes just ~0.2s longer to do both functions in parallel than the slower of the two functions running entirely separately.
Hope that helps someone!
Coincidentally, I've worked on a similar problem, and I have created a python library (more of a thin wrapper) for reading videos. The library is called mydia.
The library does not use OpenCV. It uses FFmpeg as the backend for reading and processing videos.
mydia supports custom frame selection, frame resizing, grayscale conversion and much more. The documentation can be viewed here
So, if you want to select N frames per second (where N = 1 in your case), the following code would do it:
import numpy as np
from mydia import Videos
video_path = "path/to/video"
def select_frames(total_frames, num_frames, fps, *args):
"""This function will return the indices of the frames to be captured"""
N = 1
t = np.arange(total_frames)
f = np.arange(num_frames)
mask = np.resize(f, total_frames)
return t[mask < N][:num_frames].tolist()
# Let's assume that the duration of your video is 120 seconds
# and you want 1 frame for each second
# (therefore, setting `num_frames` to 120)
reader = Videos(num_frames=120, mode=select_frames)
video = reader.read(video_path) # A video tensor/array
The best part is that internally, only those frames that are required are read, and therefore the process is much faster (which is what I believe you are looking for).
The installation of mydia is extremely simple and can be viewed here.
This might have a slight learning curve, but I believe that it is exactly what you are looking for.
Moreover, if you have multiple videos, you could use multiple workers for reading them in parallel. For instance:
from mydia import Videos
path = "path/to/video"
reader = Videos()
video = reader.read(path, workers=4)
Depending on your CPU, this could give you a significant speed-up.
Hope this helps !!