How to download a file from a url using AWS lambda with node.js code

Question

I am having a hard time to write code for my lambda function. This function has to download files into /tmp directory but instead of doing that it creates files with 0 size. The output looks like that:

2020-07-13T21:28:09.694-05:00 2020-07-14T02:28:09.681Z a914d6fa-0b29-4c66-8cc5-95095575be7b INFO /tmp/7za123.zip

2020-07-13T21:28:09.994-05:00 2020-07-14T02:28:09.975Z a914d6fa-0b29-4c66-8cc5-95095575be7b INFO /tmp/7za321.zip

2020-07-13T21:28:10.014-05:00 2020-07-14T02:28:09.995Z a914d6fa-0b29-4c66-8cc5-95095575be7b INFO /tmp/7za920.zip

2020-07-13T21:28:10.016-05:00 2020-07-14T02:28:10.016Z a914d6fa-0b29-4c66-8cc5-95095575be7b INFO 0

2020-07-13T21:28:10.017-05:00 2020-07-14T02:28:10.017Z a914d6fa-0b29-4c66-8cc5-95095575be7b INFO 0

Could you help me to fix it?

      var AWS = require('aws-sdk')
      var response = require('cfn-response')
      var fs = require('fs')
      var path = require('path')
      var https = require('https')
      var urlmod = require('url')
      var {promisify} = require('util')
      var stream = require('stream')
      var finished = promisify(stream.finished)
      var s3 = new AWS.S3()

      function getFilesizeInBytes(filename) {
        var stats = fs.statSync(filename)
        var fileSizeInBytes = stats["size"]
        return fileSizeInBytes
      }

      download = async function (filename, url) {
        console.log(filename)
        file = fs.createWriteStream(filename)
        request = https.get(url, function(resp) {
          resp.pipe(file)
        })
        await finished(file)
      }


      exports.lambda_handler = (event, context, callback) => {
        var s3buketname = event.ResourceProperties.S3BucketName
        var urls = event.ResourceProperties.Urls

        for(let url of urls) {
          filename = '/tmp/' + path.basename(urlmod.parse(url, true).pathname)
          download(filename, url)
        }

        for(let url of urls) {
          filename = '/tmp/' + path.basename(urlmod.parse(url, true).pathname)
          console.log(getFilesizeInBytes(filename))
        }

        responseData = {Result: "good"}
        response.send(event, context, response.SUCCESS, responseData)
      }

Daniele Dellafiore · Accepted Answer · 2020-07-18 10:20:45Z

1

You have to wait for the Writable Stream to fire the 'finish' event.

What is happening there is that the process terminates almost immediately because it never waits for the request.pipe(file) to be actually executed.

One strategy to do that involves 2 extra steps:

use async/awaits in loops (aka ECMAScript 2018 Asynchronous Iteration), something like:

  for await (...) {
    await download(...)
  }

for each URL, wait for the Writable Stream to emit the 'finish' event, which also can be done with async/await using streams.finished() in combination with util.promisify()

const {promisify} = require('util')
const stream = require('stream')
const finished = promisify(stream.finished)
await finished(aStream)

Please note that this is a very basic approach which will result in confusing the code even more and probably every download will be executed in sequence, as every loop will wait for the pipe to finish. On top of that, error handling for individual pipe will be a mess and many many other problems. It will work, but it won't be nice.

A much cleaner approach would come in learning how to write async pipes with Highland.js. This would not be easy for beginners with that tool, as it would immediately involve using running async code with sequence and probably also fork/merge the pipes. If you are interested I can elaborate further on that.

edited Jul 18, 2020 at 10:20

answered Jul 14, 2020 at 7:06

Daniele Dellafiore

1,9171 gold badge15 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Samuel Over a year ago

--------------- Could you correct my example to make it work with async/await using streams.finished() in combination with util.promisify().? Thank you!

Daniele Dellafiore Over a year ago

updated the answer, you can find more examples on the Node.js docs I did link.

Samuel Over a year ago

I updated my initial post regarding to your updated answer, but it did work. :(

Daniele Dellafiore Over a year ago

You applied the point 2,but point 1 is still not addressed. There is no await for the download() promise called in the for loop. You should try to run it locally and see if and when the nodejs process terminates.

Collectives™ on Stack Overflow

How to download a file from a url using AWS lambda with node.js code

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related