shell multipipe broken with multiple python scripts

Question

I am trying to get the stdout of a python script to be shell-piped in as stdin to another python script like so:

find ~/test -name "*.txt" | python my_multitail.py | python line_parser.py

It should print an output but nothing comes out of it.

Please note that this works:

find ~/test -name "*.txt" | python my_multitail.py | cat

And this works too:

echo "bla" | python line_parser.py

my_multitail.py prints out the new content of the .txt files:

from multitail import multitail
import sys

filenames = sys.stdin.readlines()
# we get rid of the trailing '\n'
for index, filename in enumerate(filenames):
  filenames[index] = filename.rstrip('\n')

for fn, line in multitail(filenames):
   print '%s: %s' % (fn, line),
   sys.stdout.flush()

When a new line is added to the .txt file ("hehe") then my_multitail.py prints:

/home/me/test2.txt: hehe

line_parser.py simply prints out what it gets on stdin:

import sys

for line in sys.stdin:
  print "line=", line

There is something I must be missing. Please community help me :)

Does either script do any kind of filtering that, when combined, filters all output? It's hard to give an answer without know what each Python script does. — chepner
– chepner, Commented Jul 28, 2016 at 1:07
Maybe line_parser.py doesn't like words that end .txt? Without more information, we really can't help. There's no obvious reason for a problem, but that only means you've not yet shown the relevant information. — Jonathan Leffler
– Jonathan Leffler, Commented Jul 28, 2016 at 1:20
It's a bit confusing that you seem to have two different multitail modules. One is the script you're running and the other is something you're importing a function from. — Blckknght
– Blckknght, Commented Jul 28, 2016 at 2:44
The problem is buffering: the data is being piped, and the output of the commands is buffered because the output is not an 'interactive device'. That means you have to wait for enough material to fill a buffer in the output of the first (Python) command before the second one sees anything. There probably is a way to force the outputs through with some flush operation in the various commands (especially the multitail one). And there'll be the problem that FatalError diagnosed too. — Jonathan Leffler
– Jonathan Leffler, Commented Jul 28, 2016 at 4:24

FatalError · Accepted Answer · 2016-07-28 04:24:37Z

3

There's a hint if you run your line_parser.py interactively:

$ python line_parser.py 
a
b
c
line= a

line= b

line= c

Note that I hit ctrl+D to provoke an EOF after entering the 'c'. You can see that it's slurping up all the input before it starts iterating over the lines. Since this is a pipeline and you're continuously sending output through to it, this doesn't happen and it never starts processing. You'll need to choose a different way of iterating over stdin, for example:

import sys

line = sys.stdin.readline()
while line:
    print "line=", line
    line = sys.stdin.readline()

answered Jul 28, 2016 at 4:24

FatalError

55k16 gold badges106 silver badges119 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

shell multipipe broken with multiple python scripts

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related