I found a solution to this here. Using the OP's example you basically run

stdbuf -oL /homedir/MyScript &> some_log.log

and then the buffer gets flushed after each line of output. I often combine this with nohup to run long jobs on a remote machine.

stdbuf -oL nohup /homedir/MyScript &> some_log.log

This way your process doesn't get cancelled when you log out.

It looks like you could use FIFOs for this:

mkfifo tasks

Producer:

for i in {1..10}; do echo "task$i"; done > tasks

Consumer:

while read task; do echo "received $task"; done  <tasks

With this, you don't have to take care of synchronization or deletion, and you don't waste any CPU time on polling -- if there's no data or if the producer hasn't opened the FIFO yet, the consumer will block (and thereby save CPU time). If the FIFO is full or the consumer hasn't opened the FIFO, the producer will block.

Bash doesn't buffer, but python will buffer if output isn't going to a terminal (because it's more efficient), so you'll want to flush stdout after every line.

#!/usr/bin/env python
import sys
import time
while True:
    print "hello"
    sys.stdout.flush()
    time.sleep(1)

IF one were truly wanting that data, I'd suggest attaching the gdb debugger to the python interpreter, momentarily stopping the task, calling fsync(1) (stdout), detach from it (resuming the process) and go peruse the output file.

Look in /proc/$(pidof python)/fd to see valid file descriptors. $(pidof x) returns the PID of process named 'x'.

# your python script is running merrily over there.... with some PID you've determined.
#
# load gdb
gdb
#
# attach to python interpreter (use the number returned by $(pidof python))
attach 1234
#
# force a sync within the program's world (1 = stdout, which is redirected in your example)
call fsync(1)
#
# the call SHOULD have returned 0x0, sync successful.   If you get 0xffffffff (-1), perhaps that wasn't stdout.  0=stdin, 1=stdout, 2=stderr
#
# remove our claws from poor python
detach
#
# we're done!
quit

I've used this method to change working dir's, tweak settings on the fly... many things. Alas, you can only call functions which are defined in the running program, fsync works nicely though.

(gdb command 'info functions' will list all of the functions available. Be careful though. You're operating LIVE on a process.)

There is also the command peekfd (found in psmisc package on Debian Jessie and others) which will allow you to see what's hiding in buffers of a process. Again, /proc/$(pidof python)/fd will show you valid file descriptors to give as arguments to peekfd.

If you don't remember -u for python, you can always prefix a command with stdbuf (in coreutils, already installed) to set stdin/stdout/stderr to unbuffered, line buffered or block buffered as desired:

stdbuf -i 0 -o 0 -e 0 python myscript.py > unbuffered.output

Of course, man pages are your friends, hey! perhaps an alias might be useful here too.

alias python='python -u'

Now your python always uses -u for all your command line endeavors!

The problem is that ./program -a asdf's output gets buffered when run in the pipeline, and that any output it's going to sit within the buffer until this one gets full.

You can control a command's output buffering using stdbuf; in particular, to run ./program -a asdf with an unbuffered stdout:

stdbuf -o0 ./program -a asdf

So that your buffered pipeline becomes:

stdbuf -o0 ./program -a asdf | script.sh

It's possible the (deceptively simple) unbuffer script that comes with expect may do what you're looking for.

If comands use stdio and are connected to a terminal they'll be flushed per line. Otherwise you'll need to use something like stdbuf on commands in a pipe line http://www.pixelbeat.org/programming/stdio_buffering/

tl;dr: instead of printf ... try to put to the script stdbuf -o0 printf .., or stdbuf -oL printf ...

The caching is handled by the libc. You can use the stdbuf command to change the buffer size:

stdbuf -o0 ./my-script.sh > file.txt &

-o0 sets the buffer size for stdout to 0. Probably you also want -e0 for stderr.

This thread on nonblocking I/O in bash might help.

It suggests using stty and dd.

Or you could use the bash read builtin with the -t 0 option.

# do your stuff

# discard rest of input before exiting
while read -t 0 notused; do
   read input
   echo "ignoring $input"
done

If you only want to do it if the user is at a terminal, try this:

# if we are at a terminal, discard rest of input before exiting
if test -t 0; then
    while read -t 0 notused; do
       read input
       echo "ignoring $input"
    done
fi

gdb -p PID -batch -ex 'p fflush(stdout)'

As with any debugging and hacking, YMMV.