From DynamoDB into SPICE (Incremental refresh)

Hey Quicksight community! It’s a pleasure to meet you all.

I hope this post finds you well :slight_smile:

What do I need:
I have DynamoDB table, which grows about 3-5 GB per month. The growth can increase in the future, though. I need this data to be available in AWS Quicksight SPICE (incremental refresh) for my dashboards.

What I’ve tried:
I’ve used Athena to query DynamoDB and deliver the data straight into SPICE. The thing was that with significant amounts of data (approx 3 GB. table), it simply doesn’t work - I’m getting a lambda timeout error.

Could you please briefly describe an example of data pipeline that would work in my case and won’t be very expensive? Considering that I need SPICE incremental refresh to work.

Thanks for your time reading this post and helping me.

You can use the GLUE ETL. this video may help you.

Incremental refresh work fine with huge dataset.

Naveed Ali

1 Like

Can you use Dynamostreams to write the data to S3, and then query the data using Athena?

Alternatively you could write a task to export dynamo to s3 every day and then use Athena on top of this.

You can write a Glue task that separates the data by date, and then use this column to only pull in the latest data for your incremental refresh.

1 Like