Trying to create ingestions for several datasets. There are hundreds of clients and about 60+ datasets. I have a nightly process that create ingestions(25k datasets) and ran into errors on throttling exceptions. The dataset IDs are prefixed with client identifier. Is there a way to create ingestions based on the dataset ID prefix or some kind of batch way to create ingestions so I don’t have to make 25k api calls to create ingestions?
Hi,
I wrote some code some time ago to restart failed ingestions, maybe you can get some ideas based on this?
Hope this helps!
1 Like
Thanks for that article! I can use that for something else but my current issue is about overcoming the throttling exception.
Hi,
What throttling are you trying to bypass? You can use the retry concept to exponential backout and automatic retry the failed jobs.
The throttling exception returned by the API. For example, creating ingestions for 500 clients which has 100 datasets per client. Ingestions are created per dataset which is one api call. So that will be 50K api calls. That will trigger a throttling exception. So I was wondering since the dataset ids has prefix of the client ids, is there an api call that I can create ingestions by batch based on the dataset id prefix? (So with that batch capability, I can minimize my api call to 500 only). So is that possible or is there other way I can minimize my api calls or can I increase the quota limit for quicksight api calls?
@andres007 do you know of an api call that i can creation ingestions by batch based on the id prefix?
Nothing I can think off directly from what I see on the APIs.
Are you using scheduled Lambdas to start the ingestions? Maybe use a StepFunction to call the lambda and perform some kind of looping / wait mechanism with the list_dataset paginator so you get only a specific number of ingestions started in a specific amount of time.
Hope this helps!
That won’t really help in the sense of minimizing the api calls. Anyways, thanks for your input!
Hi @alltej,
It’s been awhile since last communication on this thread, did you have any additional questions?
If we do not hear back within the next 3 business days, I’ll go ahead and close out this topic.
Thank you!
Hi @alltej,
Since we haven’t heard back, I’ll close out this topic. However, if you have any additional questions, feel free to create a new post in the community and link this discussion for relevant information if needed.
Thank you!