How to disable stdout buffer when running shell

Question

I am using Python to call a Shell script with

def run_command(cmd):
    print "Start to run: " + cmd
    run = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while True:
        line = run.stdout.readline().decode()[:-1]
        if line == '' and run.poll() is not None:
            break
        print line # print the log from shell
    recode = run.returncode
    if recode != 0:
        raise Exception("Error occurs!")

    print "End to run: " + cmd

Then I run

run_command("sh /home/tome/a.sh")

I notice the console output from a.sh is not in real time, looks like that there is a buffer for the stdout and when the stdout buffer is full, then the output is printed out.

I would ask how to disable the shell stdout buffer in my script a.sh

Thanks!

ShadowRanger · Accepted Answer · 2019-09-11 15:40:17Z

5

The buffering in question would largely be a problem on the script's side, not the Python side; while Python would buffer the reads, it wouldn't block unless the buffer was emptied and there was nothing available to read.

So really, you need to disable buffering in the script itself. Adding stdbuf -oL (or -o0 for completely unbuffered, but line buffering should cover you since you read by line as well) to your commands should help in some cases (where the programs don't adjust their own buffering internally).

If you're seeing this behavior only by looking at Python's output, be aware that Python itself can buffer output as well. You can disable this by passing -u when running Python, or setting the environment variable PYTHONUNBUFFERED=1 before running it, or from within a script, you can manually call sys.stdout.flush() after any writes (direct, or implicit via print) to stdout. On modern Python, print takes an argument to force a flush after printing, but since you're on Python 2.x, that's not an option.

edited Sep 11, 2019 at 15:40

answered Sep 7, 2019 at 0:48

ShadowRanger

158k12 gold badges221 silver badges316 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Tom Over a year ago

Thanks @shadowranger for the helpful answer, let me try.

Tom Over a year ago

That's too bad.. When I set the command as stdbuf -o0 sh abc.sh, the output is still buffered. Not sure where the problem is.

ShadowRanger Over a year ago

@Tom: Whether stdbuf works depends on the actual programs being run; if they adjust their own buffering internally, at runtime, then stdbuf's changes get undone. If they don't use standard streams, the changes don't affect their I/O at all. The stdbuf man page has a note on this, mentioning tee, dd and cat as being specific offenders, but those are just limited examples.

Tom Over a year ago

Thanks @hadowranger, When I run the python(which is calling shell) directy, it will output in real time, but when I run it in our scheduler system, then the output is buffered, I will investigite more to find out how to make it work.

Tom Over a year ago

When I am running python in interactive mode, then the buffer is disabled, but when I run python in non-interactive mode, then the buffer is not disabled, Even I have used unbuffer sh run.sh a b c,it also doesn't take effect. Not sure how to fix now.

|

Collectives™ on Stack Overflow

How to disable stdout buffer when running shell

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related