Running a shell command from Node.js without buffering output

Question

I'm trying to launch a shell command from Node.js, without redirecting that command's input and output -- just like shelling out to a command using a shell script, or using Ruby's system command. If the child process wants to write to STDOUT, I want that to go straight to the console (or get redirected, if my Node app's output was redirected).

Node doesn't seem to have any straightforward way to do this. It looks like the only way to run another process is with child_process, which always redirects the child process's input and output to pipes. I can write code to accept data from those pipes and write it to my process's STDOUT and STDERR, but if I do that, the APIs force me to sacrifice some flexibility.

I want two features:

Shell syntax. I want to be able to pipe output between commands, or run Windows batch files.
Unlimited output. If I'm shelling out to a compiler and it wants to generate megabytes of compiler warnings, I want them all to scroll across the screen (until the user gets sick of it and hits Ctrl+C).

It looks like Node wants to force me choose between those two features.

If I want an unlimited amount of output, I can use child_process.spawn and then do child.stdout.on('data', function(data) { process.stdout.write(data); }); and the same thing for stderr, and it'll happily pipe data until the cows come home. Unfortunately, spawn doesn't support shell syntax.
If I want shell syntax, I can use child_process.exec. But exec insists on buffering the child process's STDOUT and STDERR for me and giving them to me all at the end, and it limits the size of those buffers (configurable, 200K by default). I can still hook the on('data') events, if I want to see the output as it's generated, but exec will still add the data to its buffers too. When the amount of data exceeds the predefined buffer size, exec will terminate the child process.

(There's also child_process.execFile, which is the worst of both worlds from a flexibility standpoint: no shell syntax, but you still have to cap the amount of output you expect.)

Am I missing something? Is there any way to just shell out to a child process in Node, and not redirect its input and output? Something that supports shell syntax and doesn't crap out after a predefined amount of output, just like is available in shell scripts, Ruby, etc.?

same issue, 'until the cows come home' thanks, for that, made my day after 10h of work — neric
– neric, Commented Dec 22, 2016 at 20:35

Andrey Sidorov · Accepted Answer · 2012-12-14 13:24:02Z

48

You can inherit stdin/out/error streams via spawn argument so you don't need to pipe them manually:

var spawn = require('child_process').spawn;
spawn('ls', [], { stdio: 'inherit' });

Use shell for shell syntax - for bash it's -c parameter to read script from string:

var spawn = require('child_process').spawn;
var shellSyntaxCommand = 'ls -l | grep test | wc -c';
spawn('sh', ['-c', shellSyntaxCommand], { stdio: 'inherit' });

To summarise:

var spawn = require('child_process').spawn;
function shspawn(command) {
   spawn('sh', ['-c', command], { stdio: 'inherit' });
} 

shspawn('ls -l | grep test | wc -c');

answered Dec 14, 2012 at 13:24

Andrey Sidorov

25.6k5 gold badges65 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Joe White Over a year ago

That's a little simpler than having to pipe the output manually, but it still doesn't support shell syntax. Sure, I can manually choose a shell to invoke, but then my code isn't cross-platform.

Guig Over a year ago

if you want to get stdout and stderr, replace 'inherit' by 'pipe'

Skywalker13 · Accepted Answer · 2016-12-30 17:34:30Z

18

You can replace exec by spawn and use the shell syntax simply with:

const {spawn} = require ('child_process');
const cmd = 'ls -l | grep test | wc -c';
const p = spawn (cmd, [], {shell: true});

p.stdout.on ('data', (data) => {
  console.log (data.toString ());
});

The magic is just {shell: true}.

answered Dec 30, 2016 at 17:34

Skywalker13

5751 gold badge6 silver badges18 bronze badges

Comments

loganfsmyth · Accepted Answer · 2012-03-19 20:39:25Z

4

I haven't used it, but I've seen this library: https://github.com/polotek/procstreams

It you'd do this. The .out() automatically pipes to the process's stdin/out.

var $p = require('procstreams');
$p('cat lines.txt').pipe('wc -l').out();

If doesn't support shell syntax, but that's pretty trivial I think.

var command_str = "cat lines.txt | wc -l";
var cmds = command_str.split(/\s?\|\s?/);
var cmd = $p(cmds.shift());
while(cmds.length) cmd = cmd.pipe(cmds.shift());
cmd
  .out()
  .on('exit', function() {
    // Do whatever
  });

answered Mar 19, 2012 at 20:39

loganfsmyth

162k31 gold badges349 silver badges259 bronze badges

3 Comments

Joe White Over a year ago

I just tried it, and it doesn't support Windows batch files, which is really the main reason I want shell support. Lots of originally-Unix tools (e.g. Rake) are implemented using batch files on Windows, and I want to be able to invoke them from Node.

loganfsmyth Over a year ago

What do you mean by "doesn't support"? It crashes, says batch file not found, locks and executes forever?

Joe White Over a year ago

CreateProcessW: The system cannot find the file specified. Same error as child_process.spawn when you try to run a batch file, because batch files aren't executable (directly); if I try to run $p('rake').out();, it goes looking for a binary executable named rake.exe or rake.com, fails to find them, and errors out. You need the shell (cmd.exe) if you want to run batch files.

darelf · Accepted Answer · 2013-03-15 15:50:49Z

0

There's an example in the node docs for the child_process module:

Example of detaching a long-running process and redirecting its output to a file:

 var fs = require('fs'),
     spawn = require('child_process').spawn,
     out = fs.openSync('./out.log', 'a'),
     err = fs.openSync('./out.log', 'a');

 var child = spawn('prg', [], {
   detached: true,
   stdio: [ 'ignore', out, err ]
 });

 child.unref();

answered Mar 15, 2013 at 15:50

darelf

4,6692 gold badges20 silver badges11 bronze badges

2 Comments

Joe White Over a year ago

How does this answer the question? You're showing how to redirect output to a file; the question was very specifically about not redirecting output.

darelf Over a year ago

I think you're right. I obviously misread the question the first time. I apologize.

xixixao · Accepted Answer · 2023-10-17 22:43:34Z

0

The correct answer is

spawn("echo bla", { stdio: "inherit", shell: true });

This is both piping stdio to the current process and using the shell option to support any command string.

answered Oct 17, 2023 at 22:43

xixixao

5726 silver badges17 bronze badges

Collectives™ on Stack Overflow

Running a shell command from Node.js without buffering output

5 Answers 5

2 Comments

Comments

3 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

Comments

3 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related