Python : communication with c++ command line program not working when using <cstdio>

Question

I have the following python code, which is supposed to provide the intial input to a C++ program, then take its output and feed it back into it, until the program finishes execution:

comm.py

p = subprocess.Popen('test__1.exe', bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
    p.stdin.write(x)
    p.stdin.flush()
    p.stdout.flush()
    x = p.stdout.readline()
    print(x)
    if p.poll() != None:
        break

I am currently testing it with two simple C++ programs:

test__1.cpp:

#include <iostream>
using namespace std; 
int main()
{
    for( int i = 0; i < 3; ++i )
    {
        int n;
        cin >> n;
        cout << n+1 << endl; 
    }
    return 0;
}

test__2.cpp

#include <cstdio>
int main()
{
    for( int i = 0; i < 3; ++i )
    {
        int n;
        scanf("%d", &n);
        printf("%d\n", n+1);
    }
    return 0;
}

When comm.py opens test__1.exe everything works fine, but when it opens test__2.exe it hangs on the first call to readline(). Note that this problem does not occur when I feed test__2.exe the whole input before execution (i.e. initially set x = '1\n2\n3\n')

What could be causing this issue?

(Also, comm.py should be able to handle any valid C++ program, so only using iostream would not be an acceptable solution.)

EDIT: I also need the solution to work on Windows.

Mike Kinghan · Accepted Answer · 2017-01-15 12:57:36Z

2

It is caused by the fact that std::endl flushes the ostream and printf does not flush stdout, as you can see by amending test__2.cpp as follows:

#include <cstdio>
int main()
{
    for( int i = 0; i < 3; ++i )
    {
        int n;
        scanf("%d", &n);
        printf("%d\n", n+1);
        fflush(stdout);  //<-- add this
    }
    return 0;
}

You say that you want to module to work correctly with any C++ program, so you can't rely upon it flushing the standard output (or standard error) after every write.

That means you must cause the program's standard streams to be unbuffered and do so externally to the program itself. You will need to do that in comm.py.

In Linux (or other host providing GNU Core Utils), you could so by executing the program via stdbuf, e.g.

import subprocess

cmd = ['/usr/bin/stdbuf', '-i0','-o0', '-e0', './test__2']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
    p.stdin.write(x)
    x = p.stdout.readline()
    print(x)
    if p.poll() != None:
        break

which unbuffers all the standard streams. For Windows, you will need to research how do the same thing. For the time being I don't know.

edited Jan 15, 2017 at 12:57

answered Jan 15, 2017 at 12:13

Mike Kinghan

63.6k17 gold badges172 silver badges206 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

black-goat Over a year ago

This is a valid solution, but unfortunately it does not solve my problem, as I need comm.py to work with any valid C++ program. On flushing the ostream, why doesnt the line: p.stdout.flush() do the same?

Mike Kinghan Over a year ago

@black-goat Yes I was pondering that and still am. See update.

black-goat Over a year ago

Thank you for the input. Sadly I forgot to mention that I need it to work on Windows as well, but I will try to find an analogue to your suggestion.

Collectives™ on Stack Overflow

Python : communication with c++ command line program not working when using <cstdio>

comm.py

test__1.cpp:

test__2.cpp

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

comm.py

test__1.cpp:

test__2.cpp

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related