Optimizing SPICE Memory and Refresh Strategies in AWS Quicksight with Timestream Data

Question & Answer
, , , , ,
1

Hello Quicksight Community,

I’m currently facing a challenge with integrating data in Quicksight and would greatly appreciate any insights or assistance you can offer. Here’s a breakdown of the situation:

  1. Data Import and Integration: I’ve imported timestream data and associated metadata from a CSV file into a dataset for individualized analysis. When I use only the timestream data, a direct query works fine. However, issues arise when I join this with a second table, leading me to rely on SPICE.
  2. SPICE Memory Concerns: Despite having a substantial 80 GB of SPICE memory, my actual timestream data is just 1 GB. I have established an hourly schedule to refresh this data. Surprisingly, the SPICE capacity is quickly filled up, even though I expected it to be more than adequate for the 1 GB of data. This problem persists and intensifies each time I increase the capacity – it’s always filled up within a day.
  3. Refresh Process Query: This situation leads me to question the nature of the full refresh process in Quicksight. Does it stack new rows onto the existing data in the database, or does it completely clear out and replace the old data? This understanding is vital, particularly for managing data flow from Timestream efficiently.
  4. Seeking Guidance: If the full refresh indeed re-adds the entire dataset, I’m looking to find out how to limit this process. Ideally, I want the refresh to only incorporate newly added rows, avoiding unnecessary data duplication and SPICE memory consumption.

Your expertise and any advice on how to navigate this issue would be incredibly valuable. Thank you in advance for your help!

2

Hi @zarrinmehr

  1. You can estimate the size of of SPICE memory space your dataset will occupy based on the formula provided here.
  2. You can also use cloudwatch to monitor the increase is SPICE consumption after your refresh to understand the trend.
  3. Finally, there is an option to auto purchase SPICE capacity as well, based on usage.

Going through #1 and #2 will help you understand your current consumption and trend of SPICE capacity . #3 is FYI. You can choose to use it once you have understood your usage pattern and want to eliminate manual increase of SPICE capacity.