Python script does not print output as supposed

Question

I have a very simple (test) code which I'm running either from a Linux shell, or in interactive mode, and I have two different behaviours I cannot figure out the reason of.

I have a file generated by a Popen call, previously, where each line is a file path. This is the code used to generate the file:

with open('find.txt','w') as f:
find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)

(Incidentally, I was trying to build a PIPE originally, namely inputting the output of this command to a grep command, and since I wasn't successful in any way, I decided to break the problem down and just read the file paths from a file, and process them one by one. So maybe there is a common issue that is blocking me somewhere in this procedure).

Since in this second step I wasn't even able to open and process the files by opening the addresses contained in each line of the find.txt file, I just tried to print the file lines out, because for sure they're available in there:

with open('find.txt','r') as g:
for l in g.readlines():
    print(l)

Now, the interesting part:

if I paste the lines above into a python shell, everything works fine and I get my outputs as expected
if, on the other hand, I try to run python test.py, where test.py is the name of the file containing the lines above, no output appears in the shell's stdout.

I've tried sys.stdout.flush() to no avail. I've also inserted some dummy print() statements along the way: everything gets printed but what's after the g.readlines() statement.

Here's the full script I'm trying to make work (a pre-precursor of what I'm actually after, tbh).

#!/usr/bin/env python3
import subprocess
import sys

with open('find.txt','w') as f:
    find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)

print('hello')
with open('find.txt','r') as g:
    print('hello?')
    for l in g.readlines():
        print('help me!')
        print(l)


sys.stdout.flush()

output being:

{ancis:>106> python test.py
hello
hello?
{ancis:>106>

EDIT

I've quickly tried the very same lines (but without the call to find, which isn't available) on my python installation in Windows: it works as expected)

Based on that, I've tried to run the simpler code below:

print('hello')
with open('find.txt','r') as g:
    print('hello?')
    for l in g.readlines():
        print('help me!')
        print(l)


sys.stdout.flush()

as a script, in Linux - This also works w/o problems. This should mean that somehow I'm messing things up with the call to Popen... But what?

Axe319 · Accepted Answer · 2021-03-23 15:01:31Z

1

This is a race condition. Your call to

find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)

is opening another process and running your find command which takes a bit of time to fully execute. Python then continues on and reaches the reading of the file portion before the command is fully executed and the file is generated.

Want to test it out?

Add a time.sleep(1) just before the opening of the file.

Full test script:

#!/usr/bin/env python3
import subprocess
import time

with open('find.txt','w') as f:
    find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
time.sleep(1)
with open('find.txt','r') as g:
    for l in g:
        print(l)

To block until the process is complete you can use find.communicate(). With this you can also optionally set a timeout if that's something that you want.

#!/usr/bin/env python3
import subprocess

with open('find.txt','w') as f:
    find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
find.communicate()
with open('find.txt','r') as g:
    for l in g:
        print(l)

Source: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate

edited Mar 23, 2021 at 15:01

answered Mar 23, 2021 at 11:07

Axe319

4,3853 gold badges19 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Michele Ancis Over a year ago

Indeed it does! Besides waiting, is there any other way to require the control to be passed back after completion? I suppose this is probably the reason why my attempt to pass find's stdout to the next Popen, for a grep command, was returning nothing...

Axe319 Over a year ago

I added it to the answer.

Collectives™ on Stack Overflow

Python script does not print output as supposed

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related