1

I'm a beginner of Dynamodb and Dynamodb table stream. I have already created AWS Lambda and enabled DynamoDB stream with trigger that invokes my lambda for every added/updated/delete record. Now I want to perform initial sync operation for all my existing records. How can I do this?

  1. Is there any way to make all existing records in a table to be "reprocessed" and added to stream (so they can be processed by my lambda)?
  2. Do I have to write a custom script?
Improve this question

2 Answers 2

Reset to default
3

To my knowledge there is no way to do this without writing some custom script.

You could for instance write a script that reads every current item out of the table and then writes it back in overwriting itself and putting a new entry in the stream that would then be handled by your existing Lambda.

Another option is to not try and use the stream in any way for the existing items in the table. Leave the steam and Lambda as is for all future writes to the table and write a script that goes through all the existing items and processes them accordingly.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

-1

For this use case:

Is there any way to make all existing records in a table to be "reprocessed" and added to stream (so they can be processed by my lambda)?

I think, by creating another Lambda and setting the startingPosition as TRIM_HORIZON, you will be able to get all records again from the stream.

https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html#services-dynamodb-eventsourcemapping

Improve this answer

4 Comments

I don't think it works like that because Lambda stream keeps the events only for 24 hours and even TRIM_HORIZON can provide events older than that.
My answer was targeting OP’s first use case: “Is there any way to make all existing records in a table to be "reprocessed" and added to stream (so they can be processed by my lambda)?”
That's what I meant, it can't be done because the Lambda keeps the data for only 24 hours and TRIM_HORIZON does not have data older than 24 hours.
Yes, you are right. To be specific about this, here is the section from the docs: >TRIM_HORIZON - Start reading at the last (untrimmed) stream record, which is the oldest record in the shard. In DynamoDB Streams, there is a 24 hour limit on data retention. Stream records whose age exceeds this limit are subject to removal (trimming) from the stream. docs.aws.amazon.com/amazondynamodb/latest/APIReference/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.