I am using incremental refresh for my dataset of size 25M rows.
Due to the incremental refresh, my dataset goes up to ~65-70M and thus caused data size limit failure as I have a limited budget.
What I need :
I just want to delete the appended rows automatically.
I currently use a max (rank) function to filter out the data to the latest modified date but that is not enough, I need to delete these duplicate rows that are created
The solution you using is one good alternative to remove duplicates from Analysis view.
Along with incremental refresh, you can try adding full refresh schedule during end of the week or month. or when usage is low, this help cleaning up duplicates from QucikSight dataset and reduce the size.
We are working on this feature enhancement. I will mark this as feature request and share the details with product team. Thank you