Python: Capture stdout from subprocess.call

Question

I'm trying to do two things when executing a shell cmd with Python:

Capture stdout and print it as it happens
Capture stdout as a whole and process it when the cmd is complete

I looked at subprocess.check_output, but it does not have an stdout param that would allow me to print the output as it happens.

So after reading this question, I realized I may need to try a different approach.

from subprocess import Popen, PIPE

process = Popen(task_cmd, stdout = PIPE)
stdout, stderr = process.communicate()

print(stdout, stderr)

The problem with this approach is that according to the docs, Popen.communicate():

Reads data from stdout and stderr, until end-of-file is reached. Wait for process to terminate

I still cannot seem to redirect output both to stdout AND to some sort of buffer that can be parsed when the command is complete.

Ideally, I'd like something like:

# captures the process output and dumps it to stdout in realtime
stdout_capture = Something(prints_to_stdout = True)
process = Popen(task_cmd, stdout = stdout_capture)

# prints the entire output of the executed process
print(stdout_capture.complete_capture)

Is there a recommended way to accomplish this?

Possible duplicate of Retrieving the output of subprocess.call() — Random Davis
– Random Davis, Commented Mar 7, 2017 at 18:22
The other topics seem focused on simply capturing the output; whereas, I'm trying to both dump the output to stdout and capture it in its entirety for parsing on completion. I'm stuck at doing both simultaneously. — doremi
– doremi, Commented Mar 8, 2017 at 17:24

timotree · Accepted Answer · 2019-04-26 16:01:39Z

You were on the right track with using giving Popen stdout=PIPE, but you can't use .communicate() because it returns the values after execution. Instead, I suggest you read from .stdout.

The only guaranteed way to get the output the moment it's generated is to read from the pipe one character at a time. Here is my approach:

def passthrough_and_capture_output(args):
    import sys
    import subprocess

    process = subprocess.Popen(args, stdout=subprocess.PIPE, universal_newlines=True)
    # universal_newlines means that the output of the process will be interpreted as text
    capture = ""

    s = process.stdout.read(1)
    while len(s) > 0:
        sys.stdout.write(s)
        sys.stdout.flush()
        capture += s
        s = process.stdout.read(1)

    return capture

Note that reading one character at a time can incur significant overhead, so if you are alright with lagging behind a bit, I suggest that you replace the 1 in read(1) with a different number of characters to output in batches.

Jason · Accepted Answer · 2018-07-18 17:57:29Z

-2

from subprocess import check_output, CalledProcessError

def shell_command(args):
    try:
        res = check_output(args).decode()
    except CalledProcessError as e:
        res = e.output.decode()
    for r in ['\r', '\n\n']:
        res = res.replace(r, '')
    return res.strip()

answered Jul 18, 2018 at 17:57

Jason

2,18026 silver badges21 bronze badges

Collectives™ on Stack Overflow

Python: Capture stdout from subprocess.call

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related