Is there a way to increase the number of incremental refreshes that one can perform using the Boto3 QuickSight.Client.create_ingestion API? The limit seems to be around 100 per 24 hours, though I do not see it in the documentation. We have some pretty small DataSets that we would like to update frequently, but not sure if SPICE is the right place for this data, if it cannot handle the frequent updates. We would like to be able to use SPICE to reduce the latency when querying data from these DataSets. If we have to add latency (reducing frequency of incremental refreshes) before data is available, this does not seem like an acceptable tradeoff.
Hi @Bryan_Zeigler Indeed the limit for incremental refreshes via API is 100 per 24 hours. Can you tell me a little more about your use case? What is the underlying data source? How often are you trying to refresh it?
In general, SPICE is great for query performance, but is not really intended to be a ‘near real time’ data store. If you need data for example up to the minute, then I would recommend using Direct Query (not SPICE), but your performance will vary depending on the underlying source.
Hi @Jesse ,
The purpose of these smaller DataSets is to populate drop down controls with data based on certain values that have been selected. The underlying data source is Athena. We have previously used DIRECT_QUERY for querying this data, however, users were not happy with the speed at which our cascading controls were populated / updated.
With SPICE, the user experience with our controls is much better, but if we are unable to guarantee timely loading of data, it could be problematic. Are there any DataSources that could provide near-SPICE performance along with fewer restrictions on the number of refreshes / updates?
Also, are the API incremental refreshes counted differently than the scheduled incremental updates? If we use up our 100 API updates in a day, could we still count on data being updated, if we have scheduled incremental updates every 15 minutes? If so, we could provide users with some guarantee of when data would be loaded.
I see - so I assume you originally switched to SPICE because when using Direct Query specifically values in the controls only update every 24 hours (unless the user manually clicks to refresh the control). No matter that the underlying source is, if using Direct Query, this will still be the case (logic is we dont know when there are new values in the underlying data, and we dont want to ping it all day long for new values and increase traffic unnecessarily). If using SPICE then the control values refresh every time SPICE refreshes. Again, I assume you figured this all out by now based on your response 
The 100 incremental refreshes are a total of both API and UI-based schedules. We cant do 100 API based and then also set 15-min on the UI schedules - once you hit 100 total it will stop for the day.
Are you users based in a certain geography? Perhaps you do frequent refreshes during business hours and then dont run them in off-peak hours (use all 100 in a shorter window of hours).
Hi @Jesse ,
When using a DIRECT_QUERY DataSet to populate controls, I have not seen a 24 hour update limitation. Just to be sure and confirm that there is no 24 hour update limitation, I created an analysis with no visualizations, but with cascading controls based on an input parameter. Each time I refreshed the analysis and/or dashboard (not refreshing the controls specifically), there were corresponding Athena queries. As far as I can see, the way in which I am using the controls, DIRECT_QUERY would result in being able to see updated data right away. The only problem is that DIRECT_QUERY to Athena has more latency than we would like.
Also, I just tested exceeding the 100 API incremental refresh limit to see what would happen with the scheduled incremental refreshes. The scheduled incremental refreshes still executed after I’d exceeded the API rate limit. I suspected that this would be the case based on this stack overflow post about full refreshes: amazon web services - How to increase your Quicksight SPICE data refresh frequency - Stack Overflow.
So at least with the ability to perform an incremental refresh every 15 minutes once the API incremental refresh limit is exceeded, we have some guarantee that data will be loaded in a given amount of time. However, to me, it just seems like 100 API incremental refreshes is too low of a limit.
Using Direct Query, perhaps when you update the analysis or publish the dashboard it is sending a query to grab new values, or it could be that when using Show Relevant values it forces QS to send a query to see what values should be shown in the 2nd control. However I believe that once the dashboard is published and you arent re-publishing it frequently, the values are going to automatically refresh ever 24 hours (again maybe this behavior is different for Show Relevant Values though).
I understand the desire to use SPICE vs Athena for query performance. You could consider using an analytical datawarehouse like Redshift which would give better performance, but would cost more.
The product team intends to allow for more frequent SPICE refreshes in the future, but unfortunately I dont have a timeframe I can share.