Similar question to
- Dataset refreshes not starting on time
- How to refresh SPICE dataset using CreateIngestion API for incremental refresh?
Our implementation has thousands of datasets that are stored in SPICE, currently we refreshed the multiple datasets directly via Quicksight’s API. We thought that it might be a good idea that instead of triggering a refresh ourselves, we could schedule the refresh of all the datasets lets say at 1am. Now the questions would be:
- Is there an imposed out of the box schedule dataset refresh concurrency limit and how does this work? (for example if i have 50k datasets and i set them all to refresh at 1am would they be batched on a 50 by 50 basis or would the 50k datasets be triggered at once?)
- In case there is a concurrency limit or something similar can this be configured via Quicksight’s API? (a link to any documentation would be really helpful)
I’m open to feedback or other ideas regarding thousands of dataset being refreshed (particularly towards scheduling or refreshing strategies Quicksight API wise). If i have 50k+ datasets and want to refresh them on a daily basis what would be a recommended approach (we’re already taking into account incremental refresh)?
- Currently with our implementation when we trigger the multiple SPICE Dataset refreshes we were hitting a HTTP 429 Too Many Requests reason why we had to limit it to just 50 refreshes at a time.
Thanks to everyone in advance