First of all, thank you for the clarity of the question.

Q: Is it okay to use (multi)threading on application level with OpenCV?

A: Yes it is totally ok to use multithreading on application level with OpenCV unless and until you are using functions which can take advantage of multithreading such as blurring, colour space changing, here you can split the image into multiple parts and apply global functions throughout the divided part and then recombine it to give the final output.

In some functions such as Hough, pca_analysis which cannot give correct results when they are applied to divided image sections and then recombined, applying multithreading on application level to such functions may not give correct results and thus should not be done.

As πάντα ῥεῖ mentioned, your implementation of multithreading will not give you an advantage because you are joining the thread in the for loop itself. I would suggest you use promise and future objects(If you want an example of how to, let me know down in the comments, I will share the snippet.

Below answer took a lot of research, thanks for asking the question, it really helps me add info to my multithreading knowledge :)

Q: If yes, why are the time spans printed by my program above GROWING over time?

A: After a lot of research I found out that creating and destroying threads takes a lot of CPU as well as memory resources. When we initialize a thread(in your code by this line: thread t(blurSlowdown, nullptr); ) an identifier is written to the memory location to which this variable points and this identifier enables us to refer to the thread. Now in your program you are creating and destroying thread at a very high rate, now this is what happens, there is a thread pool allocated to a program through which our program can run and destroy threads, I will keep it short and let's look at the explanation below:

1. When you create a thread, this creates an identifier which points this thread.
2. When you destroy the thread, this memory is freed

BUT

3. When you again create a thread after no time the first thread is destroyed, the identifier of this new thread points to a new location(location other than the previous thread) in the thread pool.

4. After repeatedly creating and destroying a thread, the thread pool is exhausted and so CPU is forced to slow down our program cycles a bit so that the thread pool is again freed for making space for a new thread.

Intel TBB and OpenMP are very good at thread pool management so this problem may not occur while using them.

Q: Is TBB in 2019 now widely supported?

A: Yes, you can take advantages of TBB in your OpenCV program while also turning on TBB support on building OpenCV.

Here is a program for TBB implementation in medianBlur:

#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <iostream>
#include <chrono>

using namespace cv;
using namespace std;
using namespace std::chrono;

class Parallel_process : public cv::ParallelLoopBody
{

private:
    cv::Mat img;
    cv::Mat& retVal;
    int size;
    int diff;

public:
    Parallel_process(cv::Mat inputImgage, cv::Mat& outImage,
                     int sizeVal, int diffVal)
        : img(inputImgage), retVal(outImage),
          size(sizeVal), diff(diffVal)
    {
    }

    virtual void operator()(const cv::Range& range) const
    {
        for(int i = range.start; i < range.end; i++)
        {
            /* divide image in 'diff' number
               of parts and process simultaneously */

            cv::Mat in(img, cv::Rect(0, (img.rows/diff)*i,
                                     img.cols, img.rows/diff));
            cv::Mat out(retVal, cv::Rect(0, (retVal.rows/diff)*i,
                                         retVal.cols, retVal.rows/diff));

            cv::medianBlur(in, out, size);
        }
    }
};

int main()
{
    VideoCapture cap(0);

    cv::Mat img, out;

    while(1)
    {
        cap.read(img);
        out = cv::Mat::zeros(img.size(), CV_8UC3);

        // create 8 threads and use TBB
        auto start1 = high_resolution_clock::now();
        cv::parallel_for_(cv::Range(0, 8), Parallel_process(img, out, 9, 8));
        //cv::medianBlur(img, out, 9); //Uncomment to compare time w/o TBB
        auto stop1 = high_resolution_clock::now();
        auto duration1 = duration_cast<microseconds>(stop1 - start1);

        auto time_taken1 = duration1.count()/1000;
        cout << "TBB Time: " <<  time_taken1 << "ms" << endl;

        cv::imshow("image", img);
        cv::imshow("blur", out);
        cv::waitKey(1);
    }

    return 0;
}

On my machine, TBB implementation takes around 10ms and w/o TBB it takes around 40ms.

Q: If yes, what offers better performance, multithreading on the application level(if allowed) or TBB / OpenMP?

A: I would suggest using TBB/OpenMP over POSIX multithreading(pthread/thread) because TBB offers you better control over thread + better structure for writing parallel code and internally it manages pthreads. In case if you use pthreads you will have to take care of sync and safety etc in your code. But using these framework abstracts the need for handling thread which may get very complex.

Edit: I checked the comments regarding the incompatibility of image dimensions with the number of thread in which you want to divide the processing. So here is a potential workaround(haven't tested but should work), scale the image resolution to the compatible dimensions like:

If your image res is 485 x 647, scale it to 488 x 648 then pass it to Parallel_process then scale back the output to the original size of 458 x 647.

For comparison of TBB and OpenMP check this answer

Working on the assumption that the detection algorithm is CPU-intensive, you need to be using multiprocessing instead of multithreading since multiple threads will not run Python bytecode in parallel due to contention for the Global Interpreter Lock. You should also get rid of all the calls to sleep. It is also not clear when you run multiple threads or processes the way you are doing it what guarantees that the frames will be output in the correct order, that is, the processing of the second frame could complete before the processing of the first frame and get written to the detection_buffer first.

The following uses a processing pool of 6 processes (there is no need now for an implicit input queue).

from multiprocessing import Pool, Queue
import time
import cv2

# intialize global variables for the pool processes:
def init_pool(d_b):
    global detection_buffer
    detection_buffer = d_b


def detect_object(frame):
    time.sleep(1)
    detection_buffer.put(frame)


def show():
    while True:
        print("show")
        frame = detection_buffer.get()
        if frame is not None:
            cv2.imshow("Video", frame)
        else:
            break
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    return


# required for Windows:
if __name__ == "__main__":

    detection_buffer = Queue()
    # 6 workers: 1 for the show task and 5 to process frames:
    pool = Pool(6, initializer=init_pool, initargs=(detection_buffer,))
    # run the "show" task:
    show_future = pool.apply_async(show)

    cap = cv2.VideoCapture(0)

    futures = []
    while True:
        ret, frame = cap.read()
        if ret:
            f = pool.apply_async(detect_object, args=(frame,))
            futures.append(f)
            time.sleep(0.025)
        else:
            break
    # wait for all the frame-putting tasks to complete:
    for f in futures:
        f.get()
    # signal the "show" task to end by placing None in the queue
    detection_buffer.put(None)
    show_future.get()
    print("program ended")

The slick way to achieve this is not to do the thread housekeeping yourself but use a library that provides micro-parallelization.

OpenCV itself uses Intel Thread Building Blocks (TBB) for exactly this task -- running loops in parallel.

In your case, your loop has just four iterations. With C++11, you can write it down very easily using a lambda expression. In your example:

std::vector<cv::Mat> input = { m1, m2, m3, m4; }
tbb::parallel_for(size_t(0), input.size(), size_t(1), = {
    processBinary(input[i]); 
});

For this example I took code from here.

OpenCV can spawn threads when you call a function. However, all the work is performed before control is returned to the calling thread. For a number of reasons, asynchronous processing would add a substantial extra bit of complexity. (Consider, for instance: How would your program know when the computation was done?) It would also introduce some undesired overhead if the program didn't need to be asynchronous.

You can do asynchronous processing yourself with a minimal amount of effort, though, with C++11's threading API.

I think multiprocessing will work when you are running the function for multiple images at the same time.

Right now you are using a for loop and passing one image at a time to a single process.

Try the following approach and compare the time this takes with your original approach.

if __name__ == '__main__':
    p1 = Process(target=Upscale,args(file_list[0:len(file_list)//2]))
    p1.start()
    p2 = Process(target=Upscale,args(file_list[len(file_list)//2:len(file_list)]))
    p2.start()
    p1.join()
    p2.join()

Inside the Upscale function, make a for loop and loop through the images, and perform the same tasks as before.

Also, note that the function's argument is mentioned separately in "args" of the function "multiprocessing.Process".

P.S This is my first time answering, let me know if something is not clear.

For the output, you can try the following approach, I picked it up online through this https://superfastpython.com/multiprocessing-return-value-from-process/

# example of returning a variable from a process using a value
from random import random
from time import sleep
from multiprocessing import Value
from multiprocessing import Process

# function to execute in a child process
def task(variable):
    # generate some data
    data = random()
    # block, to simulate computational effort
    print(f'Generated {data}', flush=True)
    sleep(data)
    # return data via value
    variable.value = data

# protect the entry point
if __name__ == '__main__':
    # create shared variable
    variable = Value('f', 0.0)
    # create a child process process
    process = Process(target=task, args=(variable,))
    # start the process
    process.start()
    # wait for the process to finish
    process.join()
    # report return value
    print(f'Returned: {variable.value}')