From DynamoDB into SPICE (Incremental refresh)

Question & Answer
,
1

Hey Quicksight community! It’s a pleasure to meet you all.

I hope this post finds you well :slight_smile:

What do I need:
I have DynamoDB table, which grows about 3-5 GB per month. The growth can increase in the future, though. I need this data to be available in AWS Quicksight SPICE (incremental refresh) for my dashboards.

What I’ve tried:
I’ve used Athena to query DynamoDB and deliver the data straight into SPICE. The thing was that with significant amounts of data (approx 3 GB. table), it simply doesn’t work - I’m getting a lambda timeout error.

Question
Could you please briefly describe an example of data pipeline that would work in my case and won’t be very expensive? Considering that I need SPICE incremental refresh to work.

Thanks for your time reading this post and helping me.

2

You can use the GLUE ETL. this video may help you.

Incremental refresh work fine with huge dataset.

regards,
Naveed Ali

1 Like
3

Can you use Dynamostreams to write the data to S3, and then query the data using Athena?

Alternatively you could write a task to export dynamo to s3 every day and then use Athena on top of this.

You can write a Glue task that separates the data by date, and then use this column to only pull in the latest data for your incremental refresh.

1 Like