In Python 3, print can take an optional flush argument:
print("Hello, World!", flush=True)
In Python 2, after calling print, do:
import sys
sys.stdout.flush()
By default, print prints to sys.stdout (see the documentation for more about file objects).
In Python 3, print can take an optional flush argument:
print("Hello, World!", flush=True)
In Python 2, after calling print, do:
import sys
sys.stdout.flush()
By default, print prints to sys.stdout (see the documentation for more about file objects).
You can change the flushing behavior using the -u command line flag, e.g. python -u script.py.
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x see man page for details on internal buffering relating to '-u'
Here is the relevant documentation.
python - Usage of sys.stdout.flush() method - Stack Overflow
Clarifying stdout flushing in Python when \\n appears inside a single print() call (C vs CPython behavior) - Documentation - Discussions on Python.org
shell - Write Python stdout to file immediately - Unix & Linux Stack Exchange
Batched stdout handler doesn't get called when stdout is flushed
Videos
I'm writing a library to do some stuff with an API. I've put in some error handling to avoid the thing blowing up on the rare occasion when the API doesn't return properly using some code I found on stack exchange.
respi=requests.get(f"{burl}/{searchtype}/{iid}")
notdone=True
retries=0
while notdone:
try:
iinfo=json.loads(respi.text)
latlon=(iinfo['geo']['latitude'],iinfo['geo']['longitude'])
notdone=False
except Exception as e:
if retries==5:
print("Too many retries")
print("Exiting....")
sys.exit()
wait=(retries+1)**2
print(f'Something went wrong.... retrying in {wait} seconds')
sys.stdout.flush()
time.sleep(wait)
retries+=1
time.sleep(0.1)The question I have is, what does sys.stdout.flush() actually do here?
Python's standard out is buffered (meaning that it collects some of the data "written" to standard out before it writes it to the terminal). Calling sys.stdout.flush() forces it to "flush" the buffer, meaning that it will write everything in the buffer to the terminal, even if normally it would wait before doing so.
Here's some good information about (un)buffered I/O and why it's useful:
http://en.wikipedia.org/wiki/Data_buffer
Buffered vs unbuffered IO
Consider the following simple Python script:
import time
import sys
for i in range(5):
print(i),
#sys.stdout.flush()
time.sleep(1)
This is designed to print one number every second for five seconds, but if you run it as it is now (depending on your default system buffering) you may not see any output until the script completes, and then all at once you will see 0 1 2 3 4 printed to the screen.
This is because the output is being buffered, and unless you flush sys.stdout after each print you won't see the output immediately. Remove the comment from the sys.stdout.flush() line to see the difference.
This is happening because normally when process STDOUT is redirected to something other than a terminal, then the output is buffered into some OS-specific-sized buffer (perhaps 4k or 8k in many cases). Conversely, when outputting to a terminal, STDOUT will be line-buffered or not buffered at all, so you'll see output after each \n or for each character.
You can generally change the STDOUT buffering with the stdbuf utility:
stdbuf -oL python script.py > log
Now if you tail -F log, you should see each line output immediately as it is generated.
Alternatively explicit flushing of the output stream after each print should achieve the same. It looks like sys.stdout.flush() should achieve this in Python. If you are using Python 3.3 or newer, the print function also has a flush keyword that does this: print('hello', flush=True).
This should do the job:
import time, sys
for i in range(10):
print('bla')
sys.stdout.flush()
time.sleep(5)
As Python will buffer the stdout by default, here i have used sys.stdout.flush() to flush the buffer.
Another solution would be to use the -u(unbuffered) switch of python. So, the following will do too:
python -u script.py >> log