First impression is to look at YOLOv7 and Darknet. Do this in C/C++ for the greater control over memory, multi-threading, and multi-processing. For even greater speed, look into optimizing the template match with SIMD, and your hardware expense drastically reduces. Something like this for optimized video decompression: https://github.com/bsenftner/ffvideo (I wrote that, it's something like 18 ms per frame on an i9 3.4 GHz system) (Note that OpenCV's video decompression is also ffmpeg, but is not optimized, it buffers, and does not handle when a camera drops, it just hangs.) Might be doable with a single i9 workstation, no GPU required, if exploiting SIMD. Testing of one's hardware throughput would be required, of course. Not sure how cheap a camera you want to go, and FWIW it looks like more than one camera would be required. Considering the environment such a high speed sorter operates is controlled, lower expense cameras ought to work out fine, because the illumination the cameras operate is controlled. It looks like more than one camera might be required because the view of the tomatoes appears blocked by the portion of the machine that does the rejecting. That could be fixed by having multiple cameras on that same mechanism that does the rejecting, placing an additional camera where the perspective distortion of tomatoes on the edge of the frame makes it difficult to identify their 3-space position. Yeah, as long as those defects are well defined and you've got training samples of sufficient volume depicting those defects. The author of Darknet is a Redditor; I see his comments on CV posts every so often. More on reddit.com

I worked on a drone for a while that carried around NVIDIA Jetson computer. That would give you the horsepower for on-board object detection (as long as your model is light enough). The Jetson also ran ROS, and I controlled the drone with Mavros (the drone itself was running Ardupilot). More on reddit.com