I've had this same problem using openCV video capture with text-detection. It's not the webcam quality, but the fact that openCV can only show you frames as fast as you can process them in your gender detection. The solution that worked for me is to use multi-threading.
You can create a thread for the OpenCV video capture, then another thread for your image processing. The caveat: you can't magically make your image processing happen quicker without making changes the image process itself. It takes as long as it takes. What you can do is allow openCV to work on its own and send the frames to an exchange class, then allow the image processing to grab a frame and work at its own pace as CV2 continues as normal.
Here is a (shortened) version of my class for OCR image processing. You can see that in start() i'm creating a thread pointed at the ocr() process. This is where your gender identification process can go.
class OCR:
# def __init__(self, exchange: VideoStream, language=None):
def __init__(self):
self.exchange = None
# init stuff for OCR not relevant to my example, but note that it
# takes a VideoStream class called exchange which is where this class
# grabs frames to process
def start(self):
Thread(target=self.ocr, args=()).start()
return self
def set_exchange(self, video_stream):
self.exchange = video_stream
def ocr(self):
while not self.stopped:
if self.exchange is not None:
frame = self.exchange.frame
# # # OCR stuff goes here
Now here is the VideoStream class that grabs frames at its own pace, in a different thread. The image processing class (OCR) can then take these frames at it's own pace and the two don't affect each other's performance.
class VideoStream:
"""Class for CV2 video capture. The start() method will create a new
thread to read the video stream"""
def __init__(self, src=0):
self.stream = cv2.VideoCapture(src)
(self.grabbed, self.frame) = self.stream.read()
# self._boxes = None
self.stopped = False
def start(self):
Thread(target=self.get, args=()).start()
return self
def get(self):
while not self.stopped:
(self.grabbed, self.frame) = self.stream.read()
def get_video_dimensions(self):
width = self.stream.get(cv2.CAP_PROP_FRAME_WIDTH)
height = self.stream.get(cv2.CAP_PROP_FRAME_HEIGHT)
return int(width), int(height)
def stop_process(self):
self.stopped = True
Then you can do your CV2 imshow loop as you normally would.
exchange = VideoStream(0).start()
ocr = OCR().start()
ocr.set_exchange(exchange)
while True: # Begins a loop for the real-time OCR display
pressed_key = cv2.waitKey(1) & 0xFF
if pressed_key == ord('q'):
stop_stream_ocr(exchange, ocr)
break
frame = exchange.frame
cv2.imshow("Video Get Frame", frame)
cps1.increment()
Note that you can't really control what thread CV2 decides to use in its internal workings, but this approach will allow you to display your webcam at its natural fps and while image processing happens in the background.
Answer from raspberry_next on Stack OverflowHow to get faster/lighter video decoding - Android/Java - OpenCV
My OpenCV python script takes 22 minutes to finish. How can I make it significantly faster?
How to make execution speed faster (python opencv) - Stack Overflow
python 3.x - how to make cv2.videoCapture.read() faster? - Stack Overflow
This comes a bit late, but I was wondering this with my Logitech C920 HD Pro USB-camera on Ubuntu 20.04 and OpenCV. I tried to command the capture session to run Full HD @ 30 FPS but the FPS was fluctuating between 4-5 FPS.
The capture format for my camera defaulted as "YUYV 4:2:2". No matter how I tried to alter the video capture settings, OpenCV did not magically change the video format to match e.g. the desired FPS setting.
When I listed the video formats for my Logitech C920, it revealed:
ubuntu:~$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YUYV' (YUYV 4:2:2)
<clip>
Size: Discrete 1600x896
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 2304x1296
Interval: Discrete 0.500s (2.000 fps)
[1]: 'MJPG' (Motion-JPEG, compressed)
<clip>
Size: Discrete 1920x1080
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
The solution was to manually command the OpenCV capture device to use the compressed 'MJPG' format:
import numpy as np
import cv2
capture = cv2.VideoCapture(0)
W, H = 1920, 1080
capture.set(cv2.CAP_PROP_FRAME_WIDTH, W)
capture.set(cv2.CAP_PROP_FRAME_HEIGHT, H)
capture.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('M', 'J', 'P', 'G'))
capture.set(cv2.CAP_PROP_FPS, 30)
In my case, I have set camera like this (in c++ code):
// cam is cv::VideoCapture object
cam.set(cv::CAP_PROP_BUFFERSIZE, 1);
cam.set(cv::CAP_PROP_FPS, 20); // set fps before set fourcc
cam.set(cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('M', 'J', 'P', 'G'));
CAP_PROP_BUFFERSIZE doesn't matter, but CAP_PROP_FPS must be set before CAP_PROP_FOURCC.
By this setup, my camera fps increases from 5 to 20.
I'm working on an application in python using opencv for use in athletic coaching. I'd like it to run at 60fps, but I can't seem to get it to that speed. I've determined that the part that is slowing everything down is the capture process; frames aren't getting from the camera to the cpu fast enough. All testing I've done so far indicates that the processing and the rendering are not the bottleneck; it's specifically the video input that is slowing everything down.
Now, I know it is possible for the camera to run at 60fps, because when I open up the same camera in guvcview (at the same resolution), I can get a full 60fps.
I don't want to share my entire project, but here's the part that manages frame capture: https://pastebin.com/nnx32rQK. My main app calls the get_frame() function in its own thread. I'm running this on an intel NUC6CAYH (celeron CPU, 2.3 ghz), running linux.
Any help would be welcome!
EDIT: I went into more detail below, but it looks like having the cap.read() within the Feed class is a whole lot slower than just accessing cap.read() directly..... and I have no idea why
Traditional techniques for improving image analysis:
- Reduce the image to a monochrome sample.
- Reduce the range of samples, e.g. from 8-bit monochrome to 4-bit monochrome.
- Reduce the size of the image, e.g. 1024x1924 to 64x64.
- Reduce the frame rate, e.g 60fps to 5fps.
- Perform a higher level function to guess where the target area is with say a lower resolution, then perform the regular analysis on the cropped output, e.g. perform image recognition to locate the hand before determining the gesture.
Steve-o's answer is good for optimizing your code efficiency. I recommend adding some logic to monitor execution times to help you identify where to spend efforts optimizing.
OpenCV logic for time monitoring (python):
startTime = cv.getTickCount()
# your code execution
time = (cv.getTickCount() - startTime)/ cv.getTickFrequency()
Boost logic for time monitoring:
boost::posix_time::ptime start = boost::posix_time::microsec_clock::local_time();
// do something time-consuming
boost::posix_time::ptime end = boost::posix_time::microsec_clock::local_time();
boost::posix_time::time_duration timeTaken = end - start;
std::cout << timeTaken << std::endl;
How you configure your OpenCV build matters a lot in my experience. IPP isn't the only option to give you better performance. It really is worth kicking the tires on your build to get better hardware utilization.
The other areas to look at are CPU and memory utilization. If you watch your CPU and/or memory utilization, you'll probably find that 10% of your code is working hard and the rest of the time things are largely idle.
- Consider restructuring your logic as a pipeline using threads so that you can process multiple images at once (if you're tracking and need the results of previous images, you need to break up your code into multiple segments such as preprocessing/analysis and use a std::queue to buffer between them, and imshow won't work from worker threads so you'll need to push result images into a queue and imshow from the main thread)
- Consider using persistent/global objects for things like kernels/detectors that don't need to get recreated each time
- Is your throughput slowing down the longer your program runs? You may need to look at how you are handling disposing of images/variables within the main loop's scope
- Segmenting your code in functions makes it more readable, easier to benchmark, and descopes variables earlier (temporary Mat and results variables free up memory when descoped)
- If you're doing low-level processing on Mat pixels where you iterate over a large portion of the image, use a single parallel for and avoid writing
- Depending on how you are running your code, you may be able to disable debugging to get better performance
- If you're streaming and dumping frames, prefer changing the camera settings to throttle the streaming rate instead of dumping frames
- If you're converting from1 12 to 8 bits or only using a region of your image, prefer doing this at the camera hardware level
Here's an example of a parallel for loop:
cv::parallel_for_(cv::Range(0, img.rows * img.cols), &
{
for (int r = range.start; r < range.end; r++)
{
int x = r / img.rows;
int y = r % img.rows;
uchar pixelVal = img.at<uchar>(y, x);
//do work here
}
});
If you're hardware constrained (ie fully utilizing CPU and/or memory), then you need to look at priotizing your process/OS perfomance optimizations/freeing system resources/upgrading your hardware
- Increase the priority of the process to be more greedy with respect to other programs running on the computer (in linux you have nice(int inc) in unistd.h, in windows SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS) in windows.h)
- Optimize your power settings for maximum performance in general
- Disable CPU core parking
- Optimize your acquisition hardware settings (increase rx/tx buffers, etc) to offload work from your CPU
Hi guys, I have this code and the execution works, but it is a bit slow with pressing a key with the condition if red goes in white. How can I make it more instant? Should I use a gpu with cuda? How do I make use of a GPU instead of CPU with below code? The program is used to automate skill checks for this simulation website: https://www.mistersyms.com/tools/gitgud/
import cv2 import numpy as np import mss from pynput.keyboard import Controller
with mss.mss() as sct: monitor = {"top": 470, "left": 890, "width": 140, "height": 140} low_white = np.array([253, 253, 253]) high_white = np.array([255, 255, 255])
low_red = np.array([160, 0, 0])
high_red = np.array([255, 30, 30])
keyboard = Controller()
cordsw = []
while True:
img = np.array(sct.grab(monitor))
rgb_image = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
maskw = cv2.inRange(rgb_image, low_white, high_white)
maskr = cv2.inRange(rgb_image, low_red, high_red)
cordsr = []
yw, xw = np.where(maskw != 0)
yr, xr = np.where(maskr != 0)
for i in range(len(yw)):
cordsw.append([yw[i], xw[i]])
for i in range(len(yr)):
cordsr.append([yr[i], xr[i]])
for i in range(len(cordsr)):
if cordsr[i] in cordsw:
keyboard.press("space")
keyboard.release("space")
cordsw = []
break
if len(yw) == 0 and len(yr) == 0:
cordsw = []
if len(cordsr) == 0:
cordsw = []
if cv2.waitKey(25) & 0xFF == ord("q"):
break