The Popen docs warn against using Popen.stdin (etc), saying:

Warning: Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

Is there further explanation somewhere? What does this warning mean practically, for the use of process.stdin.write and .flush in the example code below? Is there another way to send input to a running process, which doesn't assume the process will exit after receiving input?

For context, I've got several examples of executables that accept input on stdin continually, processing lines as they come in. I'd like to run one, send input to the process, and do other things in python before sending more input. The recommended Popen.communicate doesn't support this requirement, since it waits for the process to close after sending input. I haven't found any way to do it.

The following trimmed-down example code works with cat; the real example executables have more significant processing of their input.

import subprocess
import time

duration=1
items=[
    'spam',
    'spam & spam',
    'spam, spam, spam, eggs & spam'
    ]

with open( 'outfile', 'a' ) as outfile:
    with subprocess.Popen(
            [ 'cat' ],
            stdin=subprocess.PIPE,
            stdout=outfile,
            stderr=subprocess.DEVNULL
            ) as process:
        for item in items:
            process.stdin.write(\
                    ('order "' + item + '"\n')\
                    .encode( 'utf-8' ) )
            process.stdin.flush()
            time.sleep( duration )

(edit: this code actually works with the example executables, so I'm wondering when / if the deadlocks will actually affect me)

There seems to be tons of conflicting information on the web about whether it is safe to use subprocess.Popen and communicate when using subprocess.PIPE. Some sites say a large output will deadlock while others say that just using communicate will be sufficient.

Can someone give me the final word? Is communicate safe enough even for really big outputs?

I tested it locally with about 10mb of output and it is fine but with this kind of thing, I am never sure if it is a local thing.

I do not need to read from the output until it is finished (otherwise, I would have to use something besides communicate)

Thanks!

I have this

subprocess.Popen.communicate(input=b"something") 

what to do so it won't block code.

I'm trying to grok subprocess.Popen to run and monitor a command line executable in the background. (Specifically, the HandbrakeCLI video converter.)

In this I have been partially successful, using the following command:

handbrake = subprocess.Popen( cmd )

Where cmd is a list of parameters. When I do this, I can do other things while it's running, poll() it, and terminate it if desired, and unless I kill it, it runs to completion exactly the way I want. The problem is that I also want to suppress the output. No problem, right?

handbrake = subprocess.Popen( cmd, stdout=subprocess.PIPE )

This works, but there's still SOME output. Now, my first thought was that the messages I was seeing were techincally ERROR messages. So I tried two different methods:

handbrake = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
handbrake = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT )

In both cases, the subprocess launches, I can do other things, but... it doesn't do anything. I can see HandbrakeCLI in the task manager, but it's not using any resources (where it *should* be using nearly 100%), and no file has been created in the target directory.

This leaves me with two questions that may or may not be related:

1. Why is redirecting stdout causing the program to do nothing?

2. Where is that other output coming from, and how do I suppress it? (Helpfully, it doesn't contain any information I need to capture.)

Hi everyone,

I am trying to use subprocess.Popen to open Powershell, and from this instance, open a secondary application (such that I can send it asynchronous Powershell commands from my Python code).

However, the subprocess.communicate() method seems to permanently close the stdin/stdout stream of the subprocess, which prevents me from sending it subsequent commands:

class _CLI:
def __init__(cls, appExe, appArgs, cmdlet, shell=True):
    cls._interface = subprocess.Popen(f"{appExe} {appArgs}", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)

def _execute(cls, cmd)
    # ONLY WORKS ONCE
    cls._interface.stdin.write(f"{cmd}\n".encode(encoding))
    output, error = self._interface.communicate()
    return output, error
# ...

max = _CLI(appExe="powershell.exe", cmd="3dsMax", shell=True)
max._execute("SOMETHING COOL")

When I create a new subprocess for each command, that seems to work, but I specifically don't want to open a new application each time I run a command:

class _CLI:
def init(cls, appExe, appArgs, cmdlet, shell=True):
    cls._interface = None
    cls.appExe = appExe
    cls.appArgs = appArgs

def _execute(cls, cmd)
    # WORKS FOR SUBSEQUENT CALLS
    cls._interface = subprocess.Popen(f"{cls.appExe} {cls.appArgs}", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell) cls._interface.stdin.write(f"{cmd}\n".encode(encoding)) 
    output, error = self._interface.communicate()
    return output, error
#...

max = _CLI(appExe="powershell.exe", cmd="3dsMax", shell=True)
max._execute("SOMETHING COOL")

It seems that the status of stdin/stdout is a protected property, and subprocess doesn't have any accessible methods to re-open it. I noticed that subprocess.execute() has a timeout parameter, and I figured that maybe it was just waiting for an unspecified timeout (and hanging indefinitely since it wasn't set). However, adjusting that value didn't seem to do anything.

If anyone has any suggestions, I'd really appreciate it!

EDIT: fixed the busted formatting