I was wondering if it is possible to have a more flexible refresh for the dataset…
Right now it is possible to do 15 minutes for incremental, but for example I would like to stop this query during the time 00-4:30 since nothing is happening during this time
you may consider using an event driven approach. Refresh can be invoked by API. So, you may trigger data refresh when the source data is ready.
Regarding the event driven approach, it is leverage other AWS services such as EventBridge to trigger the event. You may refer to below blog post. There is a CloudFormation available as well if you want to try hands-on
No, I did not try that, because I think it is not an optimal solution…there is some thing to set and additional cost…I think this is really a feature to have…