Hello Team,
Objective:
- I need to configure incremental refresh (every 15 minutes) during my office hours (9 AM to 7 PM)
- After office hours, my dataset will fully refresh once every midnight.
The midnight full refresh schedule is configured and working fine.
Problem:
- I am not able to configure the incremental refresh only during working hours.
- Currently, I have to create separate refresh schedule for each occurrence of the incremental refresh during my working hours.
- This converts to:
1 hour = 4 refresh schedules
10 hours = 40 refresh schedules
This is practically not maintainable for me or my team.
Can anyone help?
Hi @ajinkya_ghodake,
What is the impact of refreshing the dataset outside of your business hours if there are no changes in the data? As you mention this is a 15 minutes refresh schedule, this has to be setup as an incremental setup, therefore if there are no changes in the data then the refresh should finish quickly without having a major impact in your sources. Am I missing something?
Hello Andres,
Full refresh takes around 1 hour to refresh.
There are like 4 data-sets that use the same data-source.
Few of these datasets are used by offshore operational team and hence they need to be fully refreshed every hour
(Incremental not possible as we use some datasets inside these datasets as data-source)
When 2 of these data-sets refresh simultaneously, there is too much load on my rds server.
They fail.
And once they fail, it is like a domino effect.
Unless I restart my server, it wont get resolved.
Thats the story till now😓
Thank you for letting us know. I would recommend filing a case with AWS Support where we can dive into the details so that we can help you further. Here are the steps to open a support case. If your company has someone who manages your AWS account, you might not have direct access to AWS Support and will need to raise an internal ticket to your IT team or whomever manages your AWS account. They should be able to open an AWS Support case on your behalf. Hope this helps!