I did go through the documentation but due to lack of experience with reading documentation, I couldn't understand its internal working.
TIL that 20% of the runtime of my program was being dedicated to making cute little loading bars with fancy colors and emojis.
Turns out loops in Python are not that efficient, and I was putting loops where none were needed just to get nice loading bars.
hello and welcome to package of the day, the show where i type a few lines about a package i quite like and would be of use to programmers in a wide variety of contexts
today's package is called "tqdm"
A fast, extensible progress bar for Python and CLI
have you ever written a for loop that iterates over many variables and takes a long time to run?
to track its progress did you just add a print statement inside the loop indicating how far or which variable you're on, producing hundreds if not thousands of lines of text you only look at sparingly?
now you can do better. just import tqdm: from tqdm import tqdm
and wrap the structure you're for-ing in the tqdm() function like so: for i in tqdm(range(10000)):
now when you run your code, a dynamic progress bar like this one will automatically be generated when you start your loop:
76%|████████████████████████████ | 7568/10000 [00:33<00:10, 229.00it/s]
not only will it show you your progress; it'll provide you some simple metrics about how long the loop is taking and will take
try tqdm today!
The package tqdm has a module named tqdm inside it. Now to your tqdm you can
- Import all the modules in package
tqdmand use the module calledtqdmby:
import tqdm
for i in tqdm.tqdm(range(10)):
pass
OR
- Just import the
tqdmmodule of thetqdmpackage by:
from tqdm import tqdm
for i in tqdm(range(10)):
pass
The first tqdm is the name of the package or module. The second tqdm is a callable defined under that package/module. It could have been a different callable, such as trange:
from tqdm import trange
What you are basically doing is importing the tqdm callable inside the tqdm module.
The example shown in Usage of tqdm works well for me.
pbar = tqdm(["a", "b", "c", "d"])
for char in pbar:
pbar.set_description("Processing %s" % char)
Or alternatively, starting Python 3.8 which supports the walrus operator :=:
for char in (pbar := tqdm(["a", "b", "c", "d"])):
pbar.set_description(f"Processing {char}")
Other answers focus on dynamic description, but for a static description you can add a desc argument into the tqdm function.
from tqdm import tqdm
x = [5]*1000
for _ in tqdm(x, desc="Example"):
pass
Example: 100%|██████████████████████████████████| 1000/1000 [00:00<00:00, 1838800.53it/s]
Use imap instead of map, which returns an iterator of the processed values.
from multiprocessing import Pool
import tqdm
import time
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
with Pool(2) as p:
r = list(tqdm.tqdm(p.imap(_foo, range(30)), total=30))
Sorry for being late but if all you need is a concurrent map, I added this functionality in tqdm>=4.42.0:
from tqdm.contrib.concurrent import process_map # or thread_map
import time
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
r = process_map(_foo, range(0, 30), max_workers=2)
References: https://tqdm.github.io/docs/contrib.concurrent/ and https://github.com/tqdm/tqdm/blob/master/examples/parallel_bars.py
It supports max_workers and chunksize and you can also easily switch from process_map to thread_map.