Hello!
Is there a way to purge data from a dataset without running a full load on it?
I have a dataset that is reaching the max number of lines, but I don’t want to run a full load because the database may not be able to process it.
Hello @gabriel_kirst, something you could try to do would be to make a copy of the dataset and add a where clause to change the date that the data goes back to so you can reduce the total number of rows. Replace the old dataset with the copy and republish the updated dashboards, then set it back on the same incremental refresh. That will probably be the best path forward.
1 Like
Hi @gabriel_kirst - What is your data source? If it is a relational based, you can always use filters to reduce your data volume and that particular range of data only go to SPICE. I am not sure whether we can delete any rows from QS. This is something we have to check with QS team.
Regards -Sanjeeb
@gabriel_kirst I believe you have an incremental refresh setup for this dataset, and after some time your dataset has reached the max rows threshold.
I am sorry to say that there is no permanent solution for this at the moment. Automated purging is a much-needed feature for larger datasets. @Kristin is it possible to prioritize this feature or at least draw some attention?
The other workaround will be to have this incremental data file deployed on s3, only keep the files for the last n days, and load from there.
Hi @neelay. Thanks for reaching out! Yes. We can add this to our feature-request dashboard that goes to our PMs.
Currently we are doing some work on this site/on our tags, but I am bookmarking this to add the feature-request tag to this post once our work is complete.
2 Likes
@neelay and @gabriel_kirst I have now tagged this as a feature request, and it will go to our PMs as a feature request via our PFR dashboard. Thanks all! 
2 Likes