Is it possible to set a data retention period on a SQL dataset with incremental refresh enabled? Or even manually delete old data from it? I’m trying to prevent my dataset from growing too large and really only need it to retain the previous 2 weeks of data at most… It adds about 3.5M rows of data d…

Hi @Colin ! You’re able to define the time range when configuring the incremental refresh. I’ve attached the documentation and added an excerpt just below: For Window size, enter a number for size, and then choose an amount of time that you want to look back for changes.You can choose to refresh ch…

Hi Colin, I don’t think we have a way to set a data retention period in SPICE yet. If your dataset uses Custom SQL, could you specify a date criteria in the WHERE OR HAVING clause that only pulls data for the last 2 weeks? If its not a custom SQL data set, maybe include a join to another DATE dat…

Hello, I also shared the same concerns regarding the use of incremental refresh, if we keep adding new data then the data set size keep growing. Configure a window size only make sure that the data within this window size is updated but did not mean that data outside of this window size will be d…

This answer does not answer the question. Changing the window size does not delete old data but ingests new data. I suspect Quick Sight/AWS has not added this feature on purpose so that people don’t notice the data piling up along with the SPICE bill. Full refresh wipes all the data so limits the …

@Colin , @Giang_Nguyen , Currently there are no data retention possibilities when using SPICE. Using Custom sql. An example : My query always gets data from the last month [Screenshot 2023-12-05 at 13.45.32] SPICE schedules Daily loading process to get updated data every 30 minutes, I will se…

Setting data retention period on incremental refresh dataset?

Q&A

Giang_Nguyen November 30, 2023, 12:32pm 6

resurface this question, appreciate any clarification from Quick Sight team..

Optimizing processing time of Quick Sight SPICE refresh

Deleting old SPICE Data for cost optimisation

Topic		Replies	Views
Remove old data from SPICE dataset without full refresh Q&A data-source , feature-request , quick-sight , dataset , qls-spice , Business-Intelligence-Engineer	6	1285	February 23, 2024
Quick Sight Incremental Data Refreshes Q&A feature-request , quick-sight , data-preparation	4	1273	July 21, 2023
Incremental refresh & last X days Q&A quick-sight , dataset , Business-Intelligence-Engineer	3	70	August 5, 2024
Incremental refresh option not being shown Q&A data-source , feature-request , quick-sight , Business-Intelligence-Engineer	5	63	May 22, 2026
Refresh dataset with more flexible schedule Q&A error , feature-request , quick-sight	3	331	September 11, 2023

Setting data retention period on incremental refresh dataset?

Related topics