I’m trying to understand how incremental dataset refreshes work. We have the enterprise edition, and I’m trying to understand what criteria needs to be true for incremental to be displayed as an option on a dataset.
In my situation, I have multiple datasets that are set up as SPICE storage and use the same Data Source type (MySQL RDS instance). In some cases, I am able to configure incremental refresh on dataset, but in other cases the only option shown to me is a full dataset refresh.
I cannot seem to understand how/why I cannot set up incremental on all my datasets. I have a few particularly large datasets (400M+ rows) where incremental refresh would allow faster refresh of my reporting data.
Has anyone else faced this issue, or is there some clarifications the QuickSight team can provide on why certain datasets allow configuration of incremental, while others do not despite the fact that they share the same data source connection?
Thanks for the quick reply. Yes, my datasource is a MySQL RDS instance and the storage configuration is set to SPICE.
I would like to schedule incremental refresh based on the ‘Last Activity On’ Date attribute that is in my dataset. In my dataset, I am joining multiple data tables (all sourced by custom SQL queries against db views residing in the same MySQL db).
Here’s some screengrabs for added context around my situation:
QuickSight is sending a SQL query to the backend datasource for data retrieval, hence the processing depends on database.
Regarding table joins ,based on what I tested in Amazon Athena, below are examples of SQL when I refreshed data into SPICE.
Custom SQL with joins in QuickSight > The query sent to Athena