Performance issue was due to a bad compilation of dlib. Do not use pip install dlib which runs very very slowly for some reason compared to the proper compilation. I went from almost 10 seconds lag to about 2 seconds this way. So finally I didn't need multi-threading/processing, but I'm working on it to enhance the speed even more. Thanks for the help :)
Performance issue was due to a bad compilation of dlib. Do not use pip install dlib which runs very very slowly for some reason compared to the proper compilation. I went from almost 10 seconds lag to about 2 seconds this way. So finally I didn't need multi-threading/processing, but I'm working on it to enhance the speed even more. Thanks for the help :)
i tried a simplified approach like P.Ro mentioned in his answer with processes writing to an output queue but somehow the queue got locked most of the time because all the processes wrote to it at the same time. (just my guess) i probably did something wrong.
in the end i ended up using pipes.
the code is nasty. but if i was me a few hours ago. i would still be glad to find an example that actually runs without effort.
from multiprocessing import Process, Queue, Manager,Pipe
import multiprocessing
import face_recognition as fik
import cv2
import time
video_input = 0
obama_image = fik.load_image_file("obama.png")
obama_face_encoding = fik.face_encodings(obama_image)[0]
quality = 0.7
def f(id,fi,fl):
import face_recognition as fok
while True:
small_frame = fi.get()
print("running thread"+str(id))
face_locations = fok.face_locations(small_frame)
if(len(face_locations)>0):
print(face_locations)
for (top7, right7, bottom7, left7) in face_locations:
small_frame_c = small_frame[top7:bottom7, left7:right7]
fl.send(small_frame_c)
fps_var =0
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
# global megaman
with Manager() as manager:
video_capture = cv2.VideoCapture(video_input)
fi = Queue(maxsize=14)
threads = 8
proc = []
parent_p = []
thread_p = []
# procids = range(0,threads)
for t in range(0,threads):
p_t,c_t = Pipe()
parent_p.append(p_t)
thread_p.append(c_t)
print(t)
proc.append(Process(target=f, args=(t,fi,thread_p[t])))
proc[t].start()
useframe = False
frame_id = 0
while True:
# Grab a single frame of video
ret, frame = video_capture.read()
effheight, effwidth = frame.shape[:2]
if effwidth < 20:
break
# Resize frame of video to 1/4 size for faster face recognition processing
xxx = 930
yyy = 10/16 #0.4234375
small_frame = cv2.resize(frame, (xxx, int(xxx*yyy)))
if frame_id%2 == 0:
if not fi.full():
fi.put(small_frame)
print(frame_id)
cv2.imshow('Video', small_frame)
print("FPS: ", int(1.0 / (time.time() - fps_var)))
fps_var = time.time()
#GET ALL DETECTIONS
for t in range(0,threads):
if parent_p[t].poll():
small_frame_c = parent_p[t].recv()
cv2.imshow('recc', small_frame_c)
height34, width34 = small_frame_c.shape[:2]
# print fsizeee
if(width34<20):
print("face 2 small")
print(width34)
break
face_encodings_cam = fik.face_encodings(small_frame_c,[(0, width34, height34, 0)])
match = fik.compare_faces([obama_face_encoding], face_encodings_cam[0])
name = "Unknown"
if match[0]:
name = "Barack"
print(name)
break
frame_id += 1
# Hit 'q' on the keyboard to quit!
if cv2.waitKey(1) & 0xFF == ord('q'):
break
You should check out the multiprocessing library. If you have parts of you program that you can run in parallel, like processing multiple images and detecting a faces, you can simply define a process Pool. By default a pool will start as many processes simultaneously as you have processors. You can also define the pool size and fix it to 4.
e.g:
from multiprocessing import Pool
from glob import glob
def detect_face(file_name):
...
if __name__ == '__main__':
with Pool() as p:
p.map(detect_face, glob('*.jpg'))
You should also checkout sherlock, a collection of example on how to combine multiprocessing and opencv.
Limitations:
The algorithms in OpenCV itself are not easily executed in multiple processes. As OpenCV is written in C++, their execution depends highly on their implementation.
If you compile the OpenCV library with the OpenMP option active in the cmake file (it is off by default), many algorithms will use multiple cores automatically, without you needing to change anything in your source code. In my case I managed to achieve an up to 4.5x increase in performance. Yes, the Raspberry Pi 2 only has 4 cores, but if you do heavy number-crunching, a good division of the tasks can significantly reduce waiting times. For shorter algorithms don't expect much more than a 3x speedup.
Take care, with compiling the OpenMP libraries there is a bug in the libpng package. If the make halts with an error because libpng cannot find zlib, a quick and dirty solution can be that you just copy the source of zlib into the libpng source folder (or turn libpng off, if you don't need it).
Yes, you can multithread the process. Use the threading library in python 2, or thread in python 3.
Here is a simple example.
Besides your main thread,
Have one thread only operating the camera and constantly updating the latest frame, which is global in this case.
def camera_thread():
cam = cv2.VideoCapture(1)
_ret, self.image = cam.read()
cv2.imshow('camera', self.image)
A second thread can run the inference from the model using the latest frame.
An optional third could perhaps draw a bounding box over the face, or perform other operations etc.
You might need mutex locks in between 2nd and 3rd threads in this case as you can only begin drawing boxes after you obtain an output from the model. This allows your 2nd thread to begin inference of the next frame without waiting for other threads.
The above example will result in a smooth video output with the inference lagging behind slightly. If you're not sure how multithreading works, I suggest reading up on the basics first.
update. I use separate thread to capture image from camera and I don't see any speed up of FPS compare to serial... Please see my code.
import threading
import time
import cv2
import numpy as np
class myThread (threading.Thread):
def __init__(self, src):
print("thread -------------init-------------")
threading.Thread.__init__(self)
self.cap = cv2.VideoCapture(src)
self.stop = False
def run(self):
while(self.stop == False):
self.ret, self.frame = self.cap.read()
def Stop(self):
self.cap.release()
self.stop = True
def read(self):
return self.ret, self.frame
recognizer = cv2.face.LBPHFaceRecognizer_create()
recognizer.read('trainer/trainer.yml')
cascadePath = "haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascadePath);
thread = myThread(0)
thread.start()
time.sleep(1)
start = time.time()
frames = 0
font = cv2.FONT_HERSHEY_SIMPLEX
cap = cv2.VideoCapture(0)
while(True):
ret, frame = thread.read()
frame = cv2.resize(frame, (640, 480))
frames += 1
gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
faces = faceCascade.detectMultiScale(gray, 1.2, 5)
for(x,y,w,h) in faces:
# Create rectangle around the face
cv2.rectangle(frame, (x-20,y-20), (x+w+20,y+h+20), (0,255,0), 4)
Id, con = recognizer.predict(gray[y:y+h,x:x+w])
print(Id, con)
# Check the ID if exist
if(con < 60):
if(Id == 1):
Id = "HUY"
if(Id == 2):
Id = "HOA"
#If not exist, then it is Unknown
else:
#print(Id)
Id = "Unknow"
# Put text describe who is in the picture
cv2.rectangle(frame, (x-22,y-90), (x+w+22, y-22), (0,255,0), -1)
cv2.putText(frame, str(Id), (x,y-40), font, 2, (255,255,255), 3)
if cv2.waitKey(10) & 0xFF == ord('q'):
thread.Stop()
break
cv2.imshow("frame", frame)
end = time.time()
second = end - start
print("second:", + second)
print(frames/second)
cv2.destroyAllWindows()