Converting JSON Data from S3 upload, and using Lambda function to push to DynamoDB

Question

I've been working on an assignment recently and I feel like I'm very close to solving the problem I'm having, but I just can't seem to find anything that would help online. As the title states, I've got some JSON data being uploaded from a webpage into an S3 bucket. When a new S3 item is created, I want to take that data and store it in a DynamoDB table.

I'm using a Lambda function and testing with some data I've already stored in my S3 bucket. I've got the data in its key-value pairs in my console.logs but I just can't work out why it isn't actually storing the data.

On the left I have the data broken down into its key-value pair, i.e. "artist": "Elvis Presley", using JSON.parse(JSON.stringify(data)).

What I'm wondering, is how to push this data into the table.

var params = {
              Item: JSON.parse(JSON.stringify(data)),
              ReturnConsumedCapacity: "TOTAL",
              TableName: "s3-to-dynamo-s00187306"
             };
             dynamo.putItem(params, dynamoResultCallback);

The above code is what I've been trying to use but it's giving me a timeout error. If I bump up the allowed time then I receive a different error relating to a missing partition key in the item, even though my partition key matches with one of the key values in every item.

Really stumped here, any advice is appreciated, thanks in advance.

[edit]

So I used what someone suggested below, the dynamo-db converter, and have some logs which provide some insight into what's going on.

I've now got the data in the correct format for dynamo-db, and each item is parsed correctly as far as I can tell.

As for what dynamo represents, I'm not 100% so I'm going to add a screenshot of its declaration at the top of my code. I think it's the doc client?

[edit 2]

So my "_class" values are all the exact same, might try changing the partition key to title instead? (nevermind this didn't work)

Share your data payload, and also your DynamoDb schema, are you using sort keys, and provide in a better way the errors that you are receiving — valdeci
– valdeci, Commented Mar 7, 2022 at 14:56
Also, is dynamo the low-level DynamoDB client or the Document Client? — jarmod
– jarmod, Commented Mar 7, 2022 at 15:28

Youcef LAIDANI · Accepted Answer · 2022-03-07 15:02:36Z

1

JSON.stringify(data) return a json format that not match with Dynamodb format, Dynamodb are waiting a format like this:

Item: {
    'CUSTOMER_ID' : {N: '001'},
    'CUSTOMER_NAME' : {S: 'Richard Roe'}
}

As you see, the syntax is not the same, I think you need to use another library, maybe dynamo-converters, or look at NodeJs Aws SDK maybe there is a method that can do this.

answered Mar 7, 2022 at 15:02

Youcef LAIDANI

60.3k21 gold badges111 silver badges178 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jarmod Over a year ago

Note: the data is being stringified and then parsed.

Collectives™ on Stack Overflow

Converting JSON Data from S3 upload, and using Lambda function to push to DynamoDB

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related