I have a SPICE dataset. I made a small change to it, updating one calculated field formula without changing its name or the folder structure, and clicked “Save and Publish”.
The dataset is big and takes a couple of hours to refresh. I noticed that:
1- At first, the dataset became unavailable in analyses/dashboards for some time.
2- Then it became available before the refresh finished, but the numbers kept changing (increasing) as more rows were ingested, which was quite confusing!
By the way, the dataset is connected to an Athena table (with extra calculated fields), and this table wasn’t changed.
Can someone explain this?
Hi @saadnayef
From what you described the change in the formula and then a Save and Publish requires a full dataset refresh. Since SPICE thinks there is a structural change to the dataset it does not retain the old values and completely refreshes the dataset.
That is the reason for you see nothing initially and then as rows get added you see things changing. This is true for the first update or an update when you make a structural change to the dataset.
However, when you do incremental refreshes this should not happen since a previous snapshot exists. Only when the complete data is refreshed the change shows up.
Hi @saadnayef,
Just checking in to see if Giri’s reply was helpful, or if you were able to find your own answer in the meantime. Please help the community by marking this answer as “Solution” or following up in general within the next 3 business days!
Thanks!
I understand now that changing a formula will trigger a refresh. But I believe that a refresh should not incrementally change data, especially in my case, where my dataset takes a long time to refresh. Data should stay the same until it is completely updated.