Keeping a rolling 2 years worth of records in SPICE dataset. What is the best way?

Question & Answer
, , ,
1

I’m trying to create a dataset on SPICE (data source from Aurora) and want it to keep only the most recent 2 years’ worth of records (rolling 2 years). I also want it to refresh incrementally.

Assuming this is my custom SQL:

SELECT * 
FROM ORDERS 
WHERE date_column >= CURDATE() - INTERVAL 2 YEAR

Does doing it this way intervene with the incremental refresh in any way?

Can this be done through dataset filters?

2

hi @Ali_B

currently the incremental refresh does not support purging old data in SPICE. even if you have customer SQL with pulling rolling 2 years of data, it won’t be considered. the only way at this moment to achieve this is to do full refresh with custom SQL or dataset filter.

I will file this as feature-request.

kind regards,
Wakana

3

@Ali_B

we have raised this request,

1 Like