Scheduling daily data refresh at specific time

Question & Answer
,
1

Hi - I currently have a bunch of QuickSight datasets sourcing their data from an RDS instance (I will call it DWH), and they are refreshed daily. The RDS instance is an offline mirror of some production data which is also refreshed daily. The ETL process from production to DWH takes place at midnight and it takes few minutes to complete, depending on the table. Unfortunately, QuickSight daily refresh also takes place at midnight and sometimes it fails due to the ETL process that didn’t end yet.

At first I thought that setting up a first refresh date and time it would make all subsequent refresh taking place at the same time, but it actually didn’t work. Is there a way around this other than moving the ETL process schedule?

I also tried the route of scheduling an hourly refresh but this just increased transfer costs disproportionally.

Thank you!

2

Interesting, I would have thought the same about changing the start time. That is unfortunate. If you would be able to add a column to your dataset(s) indicating the date that the record was added, you could do an incremental refresh of only the previous day, every hour. This workaround wouldn’t be perfect as it would still refresh hourly, but at least it would only be a portion of your dataset rather than the whole thing. I would assume your dashboard would also be down from 12am-1am when then midnight refresh fails, until the next one. Unfortunately easiest solution might be to change the DWH refresh time.

image
image569×504 26.4 KB