Unload data from QuickSight Spice

Question & Answer
,
1

We are planning to replace direct query with SPICE mode for our datasets in QuickSight. Since SPICE might get pretty heavy overt the time, is it possible to unload data from SPICE? If yes, please share the related documentation.

2

Hi @sohail - You can refresh SPICE which will remove all your previous data and load the new data or you can do incremental refresh as well. See the below documentation for the same -

Regards - Sanjeeb

1 Like
3

What data do you want to unload? You can add a filter to your dataset to exclude data older than a certain date, e.g. exclude data older than 1 year.

4

Hi @Sanjeeb2022 @David_Wong, we are going to use incremental refresh. By unload i mean as data would grow over the year. We wouldn’t need old data let’s say initial 1000 rows in the spice dataset are not required. To reduce the charges we would need to unload it. Is it possible to do that, or we will have to carry forward all the data?

5

Hi @sohail - If I understood correctly you want to store incremental data and remove old data sets, I am not sure whether it is possible or not. Can you please raise a ticket to AWS Customer support team and provide these details like how you can purge the old data, to raise a ticket , please follow the link - Creating support cases and case management - AWS Support

Regards - Sanjeeb

1 Like
6

Can you not remove the old data based on a timestamp? You can put a WHERE clause in your SQL or use a dataset filter.

1 Like
7

Hi @David_Wong - In case of FULL load, yes where clause can be beneficial and restrict the history data. But in case of incremental how we are managing to remove old data? Any clean up process is there? I think this is an interesting observations.

Regards - Sanjeeb

8

Can you create a child dataset using the first dataset? Use incremental refresh with your parent dataset but not with the child dataset. You can then add the date filter to your child dataset.

1 Like
9

Hi @David_Wong - Thanks, However this way we are going to maintain 2 data sets and also so much records will be there in parent data set. We need to find a way, can we clean up from SPICE. I am thinking to have the cut down version at etl layer and refresh spice as full…

Regards - Sanjeeb

1 Like
10

SPICE is only a snapshot of your data. Your datalake , datawarehouse is single source of truth, so at any point you should be able to load any data into SPICE . Based on what data retention rules you have , this for example can be last 2 years only , last 5 years etc.

Interesting as it is, today you cannot delete specific interval based data in SPICE . Depending on how often you want to clean up old records in SPICE , you can trigger a full refresh . In this scenario you could use a dataset filter to restrict for example from which year you want to reload all the records. Then the incremental refresh will load new records in subsequent refreshes.

Kind regards,
Koushik

3 Likes