Traditional techniques for improving image analysis:
- Reduce the image to a monochrome sample.
- Reduce the range of samples, e.g. from 8-bit monochrome to 4-bit monochrome.
- Reduce the size of the image, e.g. 1024x1924 to 64x64.
- Reduce the frame rate, e.g 60fps to 5fps.
- Perform a higher level function to guess where the target area is with say a lower resolution, then perform the regular analysis on the cropped output, e.g. perform image recognition to locate the hand before determining the gesture.
Speed up opencv from python? - Stack Overflow
Python opencv - How can I speed up my movie - Stack Overflow
c++ - Getting better performance using OpenCV? - Stack Overflow
Speeding up frame capture in opencv
Traditional techniques for improving image analysis:
- Reduce the image to a monochrome sample.
- Reduce the range of samples, e.g. from 8-bit monochrome to 4-bit monochrome.
- Reduce the size of the image, e.g. 1024x1924 to 64x64.
- Reduce the frame rate, e.g 60fps to 5fps.
- Perform a higher level function to guess where the target area is with say a lower resolution, then perform the regular analysis on the cropped output, e.g. perform image recognition to locate the hand before determining the gesture.
Steve-o's answer is good for optimizing your code efficiency. I recommend adding some logic to monitor execution times to help you identify where to spend efforts optimizing.
OpenCV logic for time monitoring (python):
startTime = cv.getTickCount()
# your code execution
time = (cv.getTickCount() - startTime)/ cv.getTickFrequency()
Boost logic for time monitoring:
boost::posix_time::ptime start = boost::posix_time::microsec_clock::local_time();
// do something time-consuming
boost::posix_time::ptime end = boost::posix_time::microsec_clock::local_time();
boost::posix_time::time_duration timeTaken = end - start;
std::cout << timeTaken << std::endl;
How you configure your OpenCV build matters a lot in my experience. IPP isn't the only option to give you better performance. It really is worth kicking the tires on your build to get better hardware utilization.
The other areas to look at are CPU and memory utilization. If you watch your CPU and/or memory utilization, you'll probably find that 10% of your code is working hard and the rest of the time things are largely idle.
- Consider restructuring your logic as a pipeline using threads so that you can process multiple images at once (if you're tracking and need the results of previous images, you need to break up your code into multiple segments such as preprocessing/analysis and use a std::queue to buffer between them, and imshow won't work from worker threads so you'll need to push result images into a queue and imshow from the main thread)
- Consider using persistent/global objects for things like kernels/detectors that don't need to get recreated each time
- Is your throughput slowing down the longer your program runs? You may need to look at how you are handling disposing of images/variables within the main loop's scope
- Segmenting your code in functions makes it more readable, easier to benchmark, and descopes variables earlier (temporary Mat and results variables free up memory when descoped)
- If you're doing low-level processing on Mat pixels where you iterate over a large portion of the image, use a single parallel for and avoid writing
- Depending on how you are running your code, you may be able to disable debugging to get better performance
- If you're streaming and dumping frames, prefer changing the camera settings to throttle the streaming rate instead of dumping frames
- If you're converting from1 12 to 8 bits or only using a region of your image, prefer doing this at the camera hardware level
Here's an example of a parallel for loop:
cv::parallel_for_(cv::Range(0, img.rows * img.cols), &
{
for (int r = range.start; r < range.end; r++)
{
int x = r / img.rows;
int y = r % img.rows;
uchar pixelVal = img.at<uchar>(y, x);
//do work here
}
});
If you're hardware constrained (ie fully utilizing CPU and/or memory), then you need to look at priotizing your process/OS perfomance optimizations/freeing system resources/upgrading your hardware
- Increase the priority of the process to be more greedy with respect to other programs running on the computer (in linux you have nice(int inc) in unistd.h, in windows SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS) in windows.h)
- Optimize your power settings for maximum performance in general
- Disable CPU core parking
- Optimize your acquisition hardware settings (increase rx/tx buffers, etc) to offload work from your CPU
I'm working on an application in python using opencv for use in athletic coaching. I'd like it to run at 60fps, but I can't seem to get it to that speed. I've determined that the part that is slowing everything down is the capture process; frames aren't getting from the camera to the cpu fast enough. All testing I've done so far indicates that the processing and the rendering are not the bottleneck; it's specifically the video input that is slowing everything down.
Now, I know it is possible for the camera to run at 60fps, because when I open up the same camera in guvcview (at the same resolution), I can get a full 60fps.
I don't want to share my entire project, but here's the part that manages frame capture: https://pastebin.com/nnx32rQK. My main app calls the get_frame() function in its own thread. I'm running this on an intel NUC6CAYH (celeron CPU, 2.3 ghz), running linux.
Any help would be welcome!
EDIT: I went into more detail below, but it looks like having the cap.read() within the Feed class is a whole lot slower than just accessing cap.read() directly..... and I have no idea why
In the docs it is stated:
Note
This function should be followed by waitKey function which displays the image for specified milliseconds. Otherwise, it won’t display the image. For example, waitKey(0) will display the window infinitely until any keypress (it is suitable for image display). waitKey(25) will display a frame for 25 ms, after which display will be automatically closed. (If you put it in a loop to read videos, it will display the video frame-by-frame)
In cv2.waitKey(X) function X means the number of milliseconds for an image to be displayed on the screen. In your case it is set to 1, so theoretically you are able to achieve 1000 fps (frames per seconds). But frame decoding takes time in VideoCapture object and limits your framerate. To change the playback speed you need to declare variable and use it as a parameter in waitKey function.
import cv2
cap = cv2.VideoCapture('video.mp4')
frameTime = 10 # time of each frame in ms, you can add logic to change this value.
while(cap.isOpened()):
ret, frame = cap.read()
cv2.imshow('frame',frame)
if cv2.waitKey(frameTime) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
Alternatively, as frame decoding is the most time consuming task you can move it to the second thread and use a queue of decoded frames. See this link for details.
The third approach is to separate grabbing and decoding process and simply decode every nth frame. That will result in displaying only a subset of frames from the source video but from the user perspective the video will be played faster.
import cv2
cap = cv2.VideoCapture('video.mp4')
i=0 #frame counter
frameTime = 1 # time of each frame in ms, you can add logic to change this value.
while(cap.isOpened()):
ret = cap.grab() #grab frame
i=i+1 #increment counter
if i % 3 == 0: # display only one third of the frames, you can change this parameter according to your needs
ret, frame = cap.retrieve() #decode frame
cv2.imshow('frame',frame)
if cv2.waitKey(frameTime) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
You can use ffmpeg to speed up (or slow down) your video like fast forwarding by using "presentation time stamps".
For speed up an example would be:
ffmpeg -i YOUR_INPUT_MOVIE.mp4 -vf "setpts=0.20*PTS" YOUR_OUTPUT_MOVIE.mp4
which will speed up your movie by 5x.
For slow down an example would be:
ffmpeg -i YOUR_INPUT_MOVIE.mp4 -vf "setpts=5*PTS" YOUR_OUTPUT_MOVIE.mp4
which will slow down your movie by 5x.
Note: this method will drop frames.