In Python 2.7 or Python 3
Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:
from subprocess import check_output
out = check_output(["ntpq", "-p"])
In Python 2.4-2.6
Use the communicate method.
import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()
out is what you want.
Important note about the other answers
Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].
In Python 2.7 or Python 3
Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:
from subprocess import check_output
out = check_output(["ntpq", "-p"])
In Python 2.4-2.6
Use the communicate method.
import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()
out is what you want.
Important note about the other answers
Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].
In Python 3.7+ you can use the new capture_output= keyword argument for subprocess.run:
import subprocess
p = subprocess.run(["echo", "hello world!"], capture_output=True, text=True)
assert p.stdout == 'hello world!\n'
I'm trying to grok subprocess.Popen to run and monitor a command line executable in the background. (Specifically, the HandbrakeCLI video converter.)
In this I have been partially successful, using the following command:
handbrake = subprocess.Popen( cmd )
Where cmd is a list of parameters. When I do this, I can do other things while it's running, poll() it, and terminate it if desired, and unless I kill it, it runs to completion exactly the way I want. The problem is that I also want to suppress the output. No problem, right?
handbrake = subprocess.Popen( cmd, stdout=subprocess.PIPE )
This works, but there's still SOME output. Now, my first thought was that the messages I was seeing were techincally ERROR messages. So I tried two different methods:
handbrake = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE )handbrake = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT )
In both cases, the subprocess launches, I can do other things, but... it doesn't do anything. I can see HandbrakeCLI in the task manager, but it's not using any resources (where it *should* be using nearly 100%), and no file has been created in the target directory.
This leaves me with two questions that may or may not be related:
-
Why is redirecting stdout causing the program to do nothing?
-
Where is that other output coming from, and how do I suppress it? (Helpfully, it doesn't contain any information I need to capture.)
I've been looking around for a solution to getting the live feed of stdout and stderr.
Based on a bunch of the answers I have found, the best I can come up is the following:
import subprocess
process = subprocess.Popen(command, shell=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
nextline = process.stdout.readline()
if nextline == '':
continue
print nextline.strip()But I am very unsure if this is correct or if it is even "safe". Would anyone be able to comment/verify...?
I need to run a bash command which will take hours to run and I want to watch the output on the fly - while the command is being run, NOT when the command finishes.
Below is the snippet of my code, and it does not print the outcome of subprocess while the bash command is being executed. It does print the outcome when the command has finished executing.
commands = '''
there is one command which takes hours to run
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
out, err = process.communicate(commands)
print(out)Any advice on this? Thanks
From Python 2.7 onwards , you can use - subprocess.check_output
It returns the output of the executed command back as a byte string.
Example -
>>> import subprocess
>>> s = subprocess.check_output(["echo","Hello World!"], shell=True)
>>> s
b'"Hello World!"\r\n'
I had to use shell=True on my windows for this to work, but on linux this may not be needed.
Try this:
from itertools import combinations
from subprocess import Popen, PIPE
for pair in combinations(all_ssu, 2):
out = Popen(
['blastn',
'-query', 'tmp/{0}.fna'.format(pair[0]),
'-subject', 'tmp/{0}.fna'.format(pair[1]),
'-outfmt', '6 qseqid sseqid pident'
],
stdout=PIPE
).communicate[0]
print(out)
from How can I get terminal output in python?
STDOUT is just the standard output for the program, which is the file where anything that the program prints will be written to.
If you want the outputs as a list then you can just create an empty list (l = [] or something) before the loop and do l.append(out) at the end of each iteration of the loop
If you have Python version 2.7 or later, you can use subprocess.check_output which basically does exactly what you want (it returns standard output as a string).
A simple example (Linux version; see the note):
import subprocess
print subprocess.check_output(["ping", "-c", "1", "8.8.8.8"])
Note that the ping command is using the Linux notation (-c for count). If you try this on Windows, remember to change it to -n for the same result.
As commented below, you can find a more detailed explanation in this other answer.
Output from subprocess.call() should only be redirected to files.
You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:
from subprocess import Popen, PIPE
p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.
See here for more info.
To get all stdout as a string:
from subprocess import check_output as qx
cmd = r'C:\Tools\Dvb_pid_3_0.exe'
output = qx(cmd)
To get both stdout and stderr as a single string:
from subprocess import STDOUT
output = qx(cmd, stderr=STDOUT)
To get all lines as a list:
lines = output.splitlines()
To get lines as they are being printed by the subprocess:
from subprocess import Popen, PIPE
p = Popen(cmd, stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, ''):
print line,
p.stdout.close()
if p.wait() != 0:
raise RuntimeError("%r failed, exit status: %d" % (cmd, p.returncode))
Add stderr=STDOUT to the Popen() call to merge stdout/stderr.
Note: if cmd uses block-buffering in the non-interactive mode then lines won't appear until the buffer flushes. winpexpect module might be able to get the output sooner.
To save the output to a file:
import subprocess
with open('output.txt', 'wb') as f:
subprocess.check_call(cmd, stdout=f)
# to read line by line
with open('output.txt') as f:
for line in f:
print line,
If cmd always requires input even an empty one; set stdin:
import os
with open(os.devnull, 'rb') as DEVNULL:
output = qx(cmd, stdin=DEVNULL) # use subprocess.DEVNULL on Python 3.3+
You could combine these solutions e.g., to merge stdout/stderr, and to save the output to a file, and to provide an empty input:
import os
from subprocess import STDOUT, check_call as x
with open(os.devnull, 'rb') as DEVNULL, open('output.txt', 'wb') as f:
x(cmd, stdin=DEVNULL, stdout=f, stderr=STDOUT)
To provide all input as a single string you could use .communicate() method:
#!/usr/bin/env python
from subprocess import Popen, PIPE
cmd = ["python", "test.py"]
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True)
stdout_text, stderr_text = p.communicate(input="1\n\n")
print("stdout: %r\nstderr: %r" % (stdout_text, stderr_text))
if p.returncode != 0:
raise RuntimeError("%r failed, status code %d" % (cmd, p.returncode))
where test.py:
print raw_input('abc')[::-1]
raw_input('press enter to exit')
If your interaction with the program is more like a conversation than you might need winpexpect module. Here's an example from pexpect docs:
# This connects to the openbsd ftp site and
# downloads the recursive directory listing.
from winpexpect import winspawn as spawn
child = spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('[email protected]')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')
To send special keys such as F3, F10 on Windows you might need SendKeys module or its pure Python implementation SendKeys-ctypes. Something like:
from SendKeys import SendKeys
SendKeys(r"""
{LWIN}
{PAUSE .25}
r
C:\Tools\Dvb_pid_3_0.exe{ENTER}
{PAUSE 1}
1{ENTER}
{PAUSE 1}
2{ENTER}
{PAUSE 1}
{F3}
{PAUSE 1}
{F10}
""")
It doesn't capture output.
The indentation of your question threw me off a bit, since Python is particular about that. Have you tried something as so:
import subprocess as s
from subprocess import Popen
import os
ps = Popen(r'C:\Tools\Dvb_pid_3_0.exe', stdin = s.PIPE,stdout = s.PIPE)
print 'pOpen done..'
(stdout, stderr) = ps.communicate()
print stdout
I think that stdout will be one single string of whatever you return from your command, so this may not be what you desire, since readline() presumes you want to view output line by line.
Would suggest poking around http://docs.python.org/library/subprocess.html for some uses that match what you are up to.
I tried asking this question a few weeks ago on StackOverflow but got no response, so I figured I'd try it here.
I am at wits end trying to find a solution to this, I've spent several hours trying to solve this and came up with nothing. The problem is fairly simple. I want to monitor stdout in **real time** from a console window (so subprocess.run and subprocess.communicate(), or anything that blocks until the procedure is finished, are not options). I am using Windows systems so anything relying on Linux can't be used. Currently using Python 3.10
First Attempt:
proc = subprocess.Popen(
['launch', 'some', 'exe', 'file'],
cwd="some/directory",
bufsize=0,
universal_newlines=True,
stdout=subprocess.PIPE,
text=True,
stdin=subprocess.PIPE,
creationflags=CREATE_NEW_CONSOLE
)
while True:
output = proc.stdout.readline()
# Check if the output contains information about hitting a breakpoint
if 'Some Text' in output:
foo(var)
break
else:
print(output)Now, this works in the sense that the command runs, and I read the output in real time, and once 'Some Text' is found, foo runs. The problem is, the new console is blank, and the output is printed to the Python window, not the new console. This is because stdout = subprocess.PIPE 'steals' the output from the console Window. This does not work for my use case because the exe file is an interactive terminal that I need the user to interact with. I do not care about having them interact with the subprocess through Python, I need to capture the output so I know when to run foo.
Other Attempts:
Some of these will be more in depth than others. I will describe what I tried and why it hasn't worked.
Echo Terminal:
What I tried: Launching a thread that runs proc and launching proc2 on another thread. The idea was to echo proc.stdout (proc2.stdin = proc.stdout), and then accept user input and write it to proc.stdin.
Why it doesn't work: Writing to stdin blocks and I was unable to even get a simple echo statement to run. I didn't pursue this super in depth because my research indicated blocking would be a severe problem I'm fighting against the whole time.
Writng to sty.stdout:
proc = subprocess.Popen(
['launch', 'some', 'exe', 'file'],
cwd="some/directory",
bufsize=0,
universal_newlines=True,
stdout=sys.stdout,
text=True,
creationflags=CREATE_NEW_CONSOLE
)What I tried: When setting stdout to sys.stdout, the process behaves as expected (outputs to the newly launched console window). My idea here was to then replace sys.stdout with a "listener" that has a write() method that writes to both the original sys.stdout and a new file stream.
Why it doesn't work: When Popen writes to sys.stdout, it doesn't call sys.stdout.write. Instead, it dumps it directly to the python.stdout file stream
Pipes:
What I tried: I tried a lot here. I tried directly accessing python sys.stdout and adding a pipe to the end of it. I've tried adding a Pipe to the start of sys.stdout and then passing it along. I've tried piping subprocess.stdout back in to the subprocess.stdin
Why it doesn't work: Honestly I've tried too much here to really go into detail, but I never even got close with any of them
Microsoft Python Libraries:
What I tried: Using Microsofts Python libraries to get the pid of the launched console and then directly write to it.
Why it doesn't work: To be honest I'm not sure, but I never made any type of progress with this method
GUI:
What I tried: This is probably the closest I've gotten to what I'm looking for. I created a super simple GUI that is supposed to mock a terminal.
Why it doesn't work: It's not that this doesn't work, per say, but that it's a lot of effort. For example, when the user presses control + c (when the exe is not launched in Python), the program pauses (not terminate). Propagating ctr+c from the GUI to the subprocess is proving to be a problem. Further, this is just one functionality of a command window, and continuing down this path seems like a waste of effort for what (seems like) should be a fairly simple task.
proc = subprocess.Popen(
['launch', 'some', 'exe', 'file'],
cwd="some/directory",
bufsize=0,
universal_newlines=True,
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
text=True,
)
while True:
output = proc.stdout.readline()
write_to_gui(output)**Final Thoughts**
I'm not sure if missing something, or if what I'm trying to do is impossible, but I feel like I should be able to do it. I'm open to any other suggestions or implementations for how to do this, but I really am at the end of my rope and haven't been able to find any resources to help with this. I'm including a rough diagram of the situation in case I wasn't clear somewhere. Thanks!
Link to Diagram
The "ugly string" is what it should be printing. Python is correctly printing out the repr(subprocess.Popen(...)), just like what it would print if you said print(open('myfile.txt')).
Furthermore, python has no knowledge of what is being output to stdout. The output you are seeing is not from python, but from the process's stdout and stderr being redirected to your terminal as spam, that is not even going through the python process. It's like you ran a program someprogram & without redirecting its stdout and stderr to /dev/null, and then tried to run another command, but you'd occasionally see spam from the program. To repeat and clarify:
<subprocess.Popen object at 0xb734b26c> <-- output of python program
brettg@underworld:~/dev$ total 52 <-- spam from your shell, not from python
drwxr-xr-x 3 brettg brettg 4096 2011-05-27 12:38 . <-- spam from your shell, not from python
drwxr-xr-x 21 brettg brettg 4096 2011-05-24 17:40 .. <-- spam from your shell, not from python
...
In order to capture stdout, you must use the .communicate() function, like so:
#!/usr/bin/python
import subprocess
output = subprocess.Popen(["ls", "-a", "-l"], stdout=subprocess.PIPE).communicate()[0]
print output
Furthermore, you never want to use shell=True, as it is a security hole (a major security hole with unsanitized inputs, a minor one with no input because it allows local attacks by modifying the shell environment). For security reasons and also to avoid bugs, you generally want to pass in a list rather than a string. If you're lazy you can do "ls -al".split(), which is frowned upon, but it would be a security hole to do something like ("ls -l %s"%unsanitizedInput).split().
See the subprocess module documentation for more information.
Here is how to get stdout and stderr from a program using the subprocess module:
from subprocess import Popen, PIPE, STDOUT
cmd = 'echo Hello World'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
print output
results:
b'Hello\r\n'
you can run commands with PowerShell and see results:
from subprocess import Popen, PIPE, STDOUT
cmd = 'powershell.exe ls'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
useful link