1

I have created an API Gateway which routes to AWS Lambda function (Node 8.10). The Lambda function makes POST request to a third party API and should return the response (status and message) back to API GATEWAY. What is happening is that I am able to successfully call third party API (its a post request so I can check that Lambda is doing its job) but when I send response back to API GATEWAY, it is not able to send the updated response.

The reason for this is that Lambda is calling callback(null, response) almost immediately , whereas response from third party API comes later and hence response object is updated later (I can confirm that through console logs). I have written one callback(null, response) inside the callback function as can be seen from attached code snippet but it seems API GATEWAY considers the earliest callback response. How can I make sure that LAMBDA function sends the updated response only. Following is the attached code:

const https = require('https');

exports.handler = async (event, context, callback) => {

  var body = JSON.parse(event.body);
  var postData = JSON.stringify(body);
  const options = {
      method: 'POST',
      hostname: app_url
      path: path_value
      port: 443,
      headers: {
          'accept': 'application/json',
          'Content-Type': 'application/json',
          'Content-Length': postData.length,
          'Authorization': auth_token_value
      }
  };

  var response = {};
  var dataStr = "";

  const req = https.request(options, (res) => {
      response.statusCode = res.statusCode;
      response.headers = res.headers;

      res.on('data', (d) => {
          dataStr += d;
      });

      res.on('end', () => {
          response.body = dataStr;
          console.log(response);
          callback(null, response);
      });
  });

  req.write(postData);
  req.end();

  console.log(response);
  callback(null, response);
}
Improve this question
2
  • Remove req.write(postData) and req.end(). I'm just guessing as, these are not required in lambda. This is not normal nodeJs server. This is serverless. Commented Sep 17, 2018 at 18:26
  • I have realised that req.end() can be removed though it does not solve the issue - it was a redundant line. I would need req.write(postData) otherwise I would not be able to post the payload. @NAVIN Commented Sep 18, 2018 at 7:21

4 Answers 4

Reset to default
1

The final working code is below. I still have to figure our on res.headers part though.

const https = require('https');

exports.handler = (event, context, callback) => {

    var body = JSON.parse(event.body);
    var postData = JSON.stringify(body);
    const options = {
        method: 'POST',
        hostname: app_url,
        path: path_value,
        port: 443,
        headers: {
            'accept': 'application/json',
            'Content-Type': 'application/json',
            'Content-Length': postData.length,
            'Authorization': auth_token_value
        }
    };

    var response = {};
    var dataStr = "";

    const req = https.request(options, (res) => {
        response.statusCode = res.statusCode;
        response.headers = {}; // TODO: should be res.headers ideally

        res.on('data', (d) => {
            dataStr += d;
        });

        res.on('end', () => {
            response.body = dataStr;
            console.log(response);
            callback(null, response);
        });
    });

    req.write(postData);
    req.end();
}
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Changes done from initial code are: 1. remove async (it was making lambda not wait for final value) 2. remove final callback line (it was sending empty response before call was finished) 3. hard code res.headers to {}. original value was somehow corrupting the response - may be it was not getting converted into proper hash which is the expectation.
0

It looks like you need to remove the last two lines (console.log and the call to the callback function), otherwise they are going to be called synchronously, before the response to your request arrives to you.

Improve this answer

7 Comments

Please first see and read about AWS lambda. callback functions are the one which send response back not res.send or other functions. last two lines are what makes AWS lambda works. I'm not down voting, But its a wrong answer.
yep. NAVIN is correct. If I remove last two lines, I get Execution failed due to configuration error: Malformed Lambda proxy response. Method completed with status: 502. Which means that somehow API GATEWAY believes that Lambda did not return the response in proper format (statusCode, body, headers).
Hey @NAVIN, you're right, but the callback function is being called in the res.on('end'... block, which executes when the response arrives. Am I missing something? Since @Vikram3891 is getting that 502 error, it may be that the response never comes from the web server and that's where the problem is.
res.on('end'.... or res function only exist in https.request. It's a callback function for https not for lambda handler. lambda handler have it's own 2 input fields, event, content, and callback like in normal express framework req, res and next.
the callback written inside res.on("end"... does get executed but it is null and void because the callback written in last line is already executed and API GATEWAY has already responded for the given API. callback function written after req.end() is called immediately and hence responds with empty response object. I am just thinking if somehow I could make the callback in last line wait till response object is populated.
|
0

Short answer: was given by AdrianT. Remove the last two lines.

Long answer: When you call http.request (in req.end()) the function call is made in asynchronous mode. This means that everything that the function receives as parameters, will be executed in some point in the future.

But just after a asynchronous call (fire and forget), the function continues the execution in the next line. In your exemple the next line is console.log and then callback(null, response). Probably this two lines are executed long time (in cpu time) before the network trip make the first part of the path to the third party server.

With the callback the event source of lambda is informed of the result of the lambda invocation. Even if the function continues his execution (waiting for the still open events in the event loop), the function return was already sent. So the event source, in this case API Gateway understand that the function already did his job and forward it to the caller.

Improve this answer

8 Comments

Agree with the flow that you have mentioned. The reason because of which I am forced to add callback... in the last line is due to some weird behavior (so far for me) from Lambda-API GATEWAY duo. If I remove this line, they start complaining with Execution failed due to configuration error: Malformed Lambda proxy response. Method completed with status: 502. It seems they do not wait till res.on("end"... is executed. But as a collateral damage of adding callback line, lambda is sending empty response object since it is not yet populated.
Two things for you to investigate: First check the format of response.body. It must be an object and maybe the third party api is using some invalid type that does not parse to an Json object. Second: check the lambda execution timeout parameter. Maybe the function is signaling a timeout before you get a chance to receive the response from third party api.
Also, check the format of response.statusCode for the same reasons explained above.
If I remove the last line, I get the error. it does not even wait till res.on("end"... (there are no logs in cloud watch). If I let last line be there, then API GETWAY gets blank response object (again confirmed from cloud watch and from API GATEWAY) though res.on("end"... prints the populated response which is in perfect shape - but then its too late by the time response object gets populated. I tried to put in wait before final callback, but somehow it caused the timeout from lambda. Moreover I cannot rely on wait, since it will add a definitive slowness in API.
I tried to use another npm module called requestify but it started giving me another issue of CSRF - not sure why CSRF was not there with https module. it seems requestify is more like browser (I may be wrong).
|
0

Since the whole chaos was due to asynchronous nature of node script and I did not have any advantage due to asynchronous way of doing stuffs here in my small lambda function, I ended up writing the same code in Python (using requests module). Now it's all working fine. But I would still be interested in knowing how can I solve my problem using node script here.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.