Save and Publish a SPICE dataset causes inconsistency

Q&A
,
1

I have a SPICE dataset. I made a small change to it, updating one calculated field formula without changing its name or the folder structure, and clicked “Save and Publish”.
The dataset is big and takes a couple of hours to refresh. I noticed that:
1- At first, the dataset became unavailable in analyses/dashboards for some time.
2- Then it became available before the refresh finished, but the numbers kept changing (increasing) as more rows were ingested, which was quite confusing!

By the way, the dataset is connected to an Athena table (with extra calculated fields), and this table wasn’t changed.
Can someone explain this?

2

Hi @saadnayef

From what you described the change in the formula and then a Save and Publish requires a full dataset refresh. Since SPICE thinks there is a structural change to the dataset it does not retain the old values and completely refreshes the dataset.

That is the reason for you see nothing initially and then as rows get added you see things changing. This is true for the first update or an update when you make a structural change to the dataset.

However, when you do incremental refreshes this should not happen since a previous snapshot exists. Only when the complete data is refreshed the change shows up.