Delete old data from spice

I am doing incremental refresh on my dataset daily with a look back window of one day so that I can have latest data being cached. My use-case is to keep data for last 30 days and anything older than that should be deleted.
My data don’t get updated for previous day so I can’t use last_update column to configure incremental refresh and I don’t want to do incremental refresh with 30 days lookup window since it is same as running full refresh and it might fail because of number of partitions(>1000000).

1 Like

Hi @priam_singh

Unfortunately there is no way to delete older records in spice.

My recommendation would be to at the right time (before the DB gets too big) do a full refresh and pull in 30 days of data to clean it up. Then on a daily basis do 1 or 2 day look back window to get the latest records. Maybe when you get to 30days of data(depending on how much data you have, longer if its not that much data) do a full refresh.

On Analysis and dashboards filter for the last 30days. (you could even setup a flag as a custom column and filter on that)

Hope this helps.

Regards,
Ramon Lopez

1 Like

Hi @Ramon_Lopez , Thanks for the reply. Looks like there is no OOTB solution for this. As you said, I am planning to schedule full refresh depending on data size.

Couldn’t you do this while importing too by using a calculated field that uses the difference between now and some datetime column?