Keeping a rolling 2 years worth of records in SPICE dataset. What is the best way?

I’m trying to create a dataset on SPICE (data source from Aurora) and want it to keep only the most recent 2 years’ worth of records (rolling 2 years). I also want it to refresh incrementally.

Assuming this is my custom SQL:

WHERE date_column >= CURDATE() - INTERVAL 2 YEAR

Does doing it this way intervene with the incremental refresh in any way?

Can this be done through dataset filters?

hi @Ali_B

currently the incremental refresh does not support purging old data in SPICE. even if you have customer SQL with pulling rolling 2 years of data, it won’t be considered. the only way at this moment to achieve this is to do full refresh with custom SQL or dataset filter.

I will file this as feature-request.

kind regards,


we have raised this request,

1 Like