0

I have the following code snippet as a Python script. I get the proper job output files with no errors. However the stdout subprocess log files (i.e. COSMOSIS_output_XXX.txt etc) don't store the expected runtime logs. Instead, these files have <_io.TextIOWrapper name=9 encoding='UTF-8'> as the only output written in the files. What is it I am doing wrong?

import subprocess
from subprocess import Popen
jobname = "cosmosis"
arg2 = "output.filename="
Vector = (
    " " + ini + " " + arg2 + COSMOSIS_PATH + os.path.splitext(ini)[0] + ".txt"
)                                                                                                                                                                                               

job3 = subprocess.Popen(
    ["cosmosis" + Vector],
    shell=True,
    text=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

file = open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file
print(job3.stdout)
file_e = open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file_e
print(job3.stderr)
try:
    outs, errs = job3.communicate(timeout=2000)                                                                                                                                      
except TimeoutExpired:
    outs, errs = job3.communicate()
    job3.kill()
file.close()
file_e.close()
Improve this question

3 Answers 3

Reset to default
1

Passing a list as the first argument with shell=True is system-dependent. I'm guessing you really mean

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
        open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
    try:
        job3 = subprocess.run(
            ["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] + ".txt"],
            stdout=file, stderr=file_e, check=True,
            timeout=2000)
    except TimeoutExpired:
        pass

There is no way for job3.stdout or job3.stderr to contain anything because we redirected them to files. (We open the file handles with binary mode so we don't need to specify a text encoding.) There is also no way for the process to return a useful result if it is killed by a timeout, and obviously no need to kill it when it was already killed.

As the subprocess documentation already tells you, you should prefer subprocess.run or one of the legacy high-level wrappers instead of Popen when you can. Perhaps see also Actual meaning of shell=True in subprocess which also explains why you want to avoid using a shell when you can.

On the other hand, if (as indicated in comments) you want the process to run in the background until completion while your Python script proceeds to perform other tasks and take out the timeout, you do need Popen.

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
        open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
    job3 = subprocess.Popen(
        ["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] + ".txt"],
        stdout=file, stderr=file_e)
...
# time passes and your script does other things ...
job3.wait()

You might want to periodically poll the job to see if it has finished.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

this is useful. My process can run between 20 secs to 2 hrs. I used timeout, because otherwise couldnot get control back from console without hitting a key. Any suggestion to fix that ?
Fix what? Avoiding a timeout? You can run a background process if you want it to run completion but allow your Python script to continue executing, in which case you need to fall back to Popen after all (but probably still avoid shell=True).
Yes exactly. If script runs for 2 hours. It should run in background, giving the control back automatically without me hitting any key. I havent found a typical example/suggestion to this. So the best I could do was this, using popen. But I am guessing this is not elegant or correct.
It's exactly how you do it. It's not terribly elegant but it is the correct solution. I updated with a brief snippet to show this, too.
1

First of all you are just opening the file, you are not reading anything from it, you are not storing the information of the file anywhere, so it will just create that <_io.TextIOWrapper name=9 encoding='UTF-8'> which is very easy reproducible:

file = open("testtextf.txt","a")
print(file)

You have to read it somehow for example with .read():

data = file.read()

Also i do not recommend using "open" unless you want to deal with files being left open if anything goes wrong in between the lines before you close it again.

I highly recommend to use "with open" instead.

Improve this answer

1 Comment

Thanks. Can you please provide a full, reproducible snippet ?
-1

Ok, the following snippet is sufficient solution to the question posted above.

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "wb") as file, open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "wb") as file_e:
    job3 = subprocess.Popen(
        ["cosmosis" + Vector],
        shell=True,
        text=True,
        stdout=file,
        stderr=file_e,
    )
Improve this answer

4 Comments

Passing a list of strings with shell=True works on Windows but not really on other platforms (well, you can make it work, but you have to know what you are doing). Probably avoid this combo.
I dont work on windows. This works very well in Linux.
The square brackets around the string are unnecessary and somewhat misleading. You should pass just a string with shell=True, or a properly split list of tokens without (try shlex.split(), or just don't glue the parts together in the first place). See also Actual meaning of shell=True in subprocess
@AyanMitra If you don’t want this answer to remain here, you can delete it yourself using the Delete link above

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.