Challenges in Managing Multiple Datasets with 15-Minute Incremental Refreshes

Question & Answer
, ,
1

Hello,

I’m facing a challenge in managing over 45 datasets that require incremental refreshes every 15 minutes. As the number of datasets continues to grow on a weekly basis, I frequently encounter the “Ingestion Superseded” error, indicating that a workflow has been replaced by another. It’s important to note that these datasets are not scheduled to refresh simultaneously. I’m wondering if SPICE is limited to handling only one dataset refresh at a time and what are the limitations of incremental refreshing?

1 Like
2

How long do your incremental refreshes take in general? Do they ever take more than 15 minutes? The only time I’ve seen the “Ingestion Superseded” error is when I have a refresh in progress and I start another one. The one in progress then fails with the “Ingestion Superseded” error and the new one starts.

2 Likes
3

Hi @David_Wong - Can we run the refresh of spice in parallel? for example i have 10 data sets, i want to refresh them in parallel, can i do that? Please advise.

Regards - Sanjeeb

4

Hi @Sanjeeb2022,
Yes, you should be able to refresh them in parallel. I personally have more than 10 datasets that I refresh at the same time every day.

2 Likes
5

Thanks @David_Wong for sharing the details.

Have a great weekend.

Regards - Sanjeeb

1 Like
6

Hi @David_Wong ,

Thanks for the response. Overall they are all under a minute, most of the datasets averaging around 20-35 seconds to refresh.

7

Hi @David_Wong ,

I got 40 Ingestion Superseded error in 2 days this weekend. How can datasets incremental refresh at the same time or in parallel without causing this when the incremental refreshes are all under a minute, not scheduled at the same time and no staff was using the platform (in the back end)?

8

Hi @Marie-LineF,

I would suggest reporting the issue to AWS support. I have multiple datasets refreshing concurrently at any given time without issues but most of them use full refresh instead of incremental. Maybe there is some limitation with incremental refresh that we’re not aware of.

2 Likes