Hello, my full dataset refresh failed due to a timeout error. My data source has a date column, and I want to set up an incremental refresh with a 1-day back frequency to append to the existing dataset. However, when I attempt to create a new schedule, I don’t see the option for incremental refresh. The only options available are time zone, start time, and frequency. My data source is a CSV file stored in an S3 bucket, and my Quick Sight account is on the Enterprise subscription
Hello @Vamsi_Prasad,
Welcome to the Quick Sight community, it is great having you here.
For S3 datasources, the incremental update option is not available.
If possible, you could access the data in S3 via Athena, and then perform an incremental update from there.
Here is a link to documentation on how to access S3 data from Athena.
Hope this helps!
1 Like
Hello @andres007 Thanks a lot for the solution.
Can I simply use AWS Athena to create a table and connect it to an S3 CSV file, and then set up an incremental connection to Quick Sight? Or should I be concerned that Athena might encounter errors loading the S3 CSV data due to data threshold limits
You can try with the data as CSV. or you can also convert the CSV to parquet where you can use compression and will make the load into SPICE faster.
Hope this helps!
1 Like