Node.js request web page

Question

I need to connect to a web page and return the status code of the page, which I've been able to achieve using http.request however the pages I need to request can take a long time, sometimes several minutes, so I'm always getting a socket hang up error.

I'm using the following code so far:

var reqPage = function(urlString, cb) {
    // Resolve the URL
    var path = url.parse(urlString);
    var req = http.request({
        host: path.hostname,
        path: path.pathname,
        port: 80,
        method: 'GET'
    });
    req.on('end', function() {
        cb.call(this, res);
    });
    req.on('error', function(e) {
        winston.error(e.message);
    });
};

What do I need to do to ensure that my application still attempts to connect to the page even if it's going to take a few minutes?

Noah · Accepted Answer · 2013-11-25 19:38:29Z

10

Use the request module and set the timeout option to an appropriate value (in milliseconds)

var request = require('request')
var url = 'http://www.google.com' // input your url here

// use a timeout value of 10 seconds
var timeoutInMilliseconds = 10*1000
var opts = {
  url: url,
  timeout: timeoutInMilliseconds
}

request(opts, function (err, res, body) {
  if (err) {
    console.dir(err)
    return
  }
  var statusCode = res.statusCode
  console.log('status code: ' + statusCode)
})

edited Nov 25, 2013 at 19:38

answered Apr 5, 2013 at 16:19

Noah

34.4k5 gold badges40 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

James Over a year ago

Perfect! Thank you. It's been requesting the page for a couple of minutes now and not given up. Cheers.

James Over a year ago

Ah, after a while I got ECONNREFUSED once I requested the second page.

Noah Over a year ago

That is probably something to do with the server. That said you could use the whilst function of the async module to retry each request until it succeeds. github.com/caolan/async#whilst

James Over a year ago

Hmm, the page it's connecting to is a PHP script running on an Apache server. The script runs set_time_limit(0); so the content is always finally delivered and if I connect directly to the page, the request completes.

Noah Over a year ago

@MaksimDmitriev fixed. Thanks for the catch

|

kberg · Accepted Answer · 2013-04-05 16:24:27Z

0

Add this if you don't want to use a higher level http client like request or superagent , then add this...

req.on("connection", function(socket){
    socket.setTimeout((1000*60*5)); //5 mins  
});

answered Apr 5, 2013 at 16:24

kberg

2,5292 gold badges21 silver badges12 bronze badges

2 Comments

James Over a year ago

Even with this I get socket hangup.

kberg Over a year ago

Are you sure that a successful connection is ever made? Have you confirmed that you got a connection and that the connection times out because of no data response? You could add logging to the "connection" event and also log on the socket "close" event. I'd also add a logging statement to the socket "error" event so see what's happening there. Given what you've described it seems likely the server you're connecting to is the one dropping the connection, in which case no client configuration will fix that. Firewalls timeouts may be plaguing you if sockets are inactive for to long.

Collectives™ on Stack Overflow

Node.js request web page

2 Answers 2

6 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related