Lambda Calls DynamoDB DocumentClient Multiple Times through While Loop

Question

Happy Thanksgiving everyone! I am writing a lambda function which calls DynamoDB and I am running into a design/implementation issue.

I have the following code structure:

exports.handler = function(event, context, callback) {
    documentClient.batchGet(getParams, function(err, data) {
        if (err) {
            console.log(err);
        } else {
         ...
              while (1) {
                    documentClient.scan(scanParams, function(err, data)    
                    {
                        if (err) {
                            console.log(err);
                        } else {
                              ...
                              if (some condition) {
                                   break;
                              }
                         }
                 }
          }
   }

I can't call break from inside the callback of a documentClient. Also I can't pass variables from inside the documentClient's callback to outside, and then break out of the while loop. I tried creating a variable in the scope of the handler and assigning it a value inside the documentClient callback function, but once the code gets out of scope of the callback, that value is erased and the variable has its original value in the handler scope.

The reason why there is a while loop is because if the table is large scanning will take more than one trial.

Also, another issue is I want to do a callback in the same scope as the handler, not within the nested blocks like the callback function of the scan documentClient. However, I can't seem to pass any data or variable from inside the nested blocks to outside.

I searched and can't find any doc's on this problem. Thanks in advance for reading.

Actually I tried using a var again and this time I could change its value from inside the callback function. However, I think the code would be cleaner if I can pass a function into the callback, rather than calling the callback in a nested manner like function(err, data) { ... }. I tried passing in a function instead of doing the nesting but it couldn't pick up the err and data returned from the documentClient call. — Sarah Guo
– Sarah Guo, Commented Nov 24, 2018 at 0:49

Itay Maman · Accepted Answer · 2018-11-25 19:56:25Z

1

Assuming you can only use Node.js 6.x (and not allowed to use newer versions) here is the answer:

// Node.js 6.x answer:

exports.handler = function (event, context, callback) {
    documentClient.batchGet(getParams, function (err, data) {
        if (err) {
            console.log(err);
        } else {
            // ...
            runScan(scanParams, function(err, scanOutput) {
                if (err) return callback(err);
                // do something with scanOutput.
            });
        }
    });
};


function runScan(scanParams, callback) {
    documentClient.scan(scanParams, function (err, data) {
        if (err) return callback(err);
        if (some_condition) {
            var scanOutput = ...;
            return callback(null, scanOutput);
        }

        // Halting condition
        if(!data.LastEvaluatedKey) {
            return callback(new Error('End of scan'));
        }

        scanParam.ExclusiveStartKey = data.LastEvaluatedKey;
        runScan(scanParams, callback);
    });
}

Key takeaways:

I moved the scanning functionality into its own function (runScan()).
Instead of using a while-loop we use recursion: runScan() calls itself - but with a different ExclusiveStartKey - to scan the next batch of items.
runScan() it an async method: it does not return value in the traditional sense. Instead it takes a callback function. If it detects an error it calls callback(err) (equivalent of throwing an exception). If it wants to "return a value" to its caller it calls callback(null, value); The caller must pass a callback function (that takes two parameters) and inspect them respectively.
The scan ends when there is no LastEvaluatedKey. In here it is reported back to the caller by passing an Error object (an exception) as the first parameter of the callback.
Finally, if you are allowed to use Node.js 8.x (or higher) you can use async/await which greatly simplify writing async code.

edited Nov 25, 2018 at 19:56

answered Nov 24, 2018 at 19:57

Itay Maman

30.8k12 gold badges93 silver badges124 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sarah Guo Over a year ago

Thanks for answering, I will try out your code structure sometime. I am using Node.js 8.x so I think I will give async / await a try as well.

Itay Maman Over a year ago

Please see my addition about the halting condition

Collectives™ on Stack Overflow

Lambda Calls DynamoDB DocumentClient Multiple Times through While Loop

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related