We run the dataset ingestions on a scheduled basis during off-hours. The dataset ingestions can sometimes run up to 2 hours for some of our customers depending on their dashboards.
We are planning to run the dataset ingestion earlier but we were not sure how it affects the dashboards while their datasets can be in a “running” status. Does it use the last dataset/ingestion that was in “COMPLETE” status while the new ingestion is still in “RUNNING” status? Is there anything we need to worry about the dashboard data and visuals while new datasets are getting ingested/running?
Hi @alltej,
There shouldn’t be any problems. Readers who access your dashboards while your ingestion is in progress will see the data from the last successful ingestion.
Are the datasets also versioned like the dashboard? I want to be able to describe a specific version of the dataset like the dashboard. Like being able to describe the version of the last completed/successful and the version that had a failed status.
Hi @David_Wong - I believe the assumption is ingestion process incremental not full. Please confirm.
Regards - Sanjeeb
In my case, this is full refresh.
1 Like
Hi @alltej - thanks for the details. I am not sure during the full refresh of SPICE, whether SPICE bring the data and then replace it or first remove the existing data and then do the refresh. If later is true, then your dashboard will not have any data during the refresh. Let’s wait for @David_Wong suggestion on this.
Regards - Sanjeeb
@alltej There is versioning for datasets. You can view the publishing history to see the older versions but I don’t see a way to use the API to get the dataset definition for an older version.
@Sanjeeb2022 There’s no impact to readers when a dataset is refreshing. Readers will continue to see data from the previous successful refresh until the current refresh finishes.
2 Likes
Thank you @David_Wong for the insight.
Regards - Sanjeeb