Hi,
I encountered an issue where we normally refresh our datasets on Tuesdays for last weeks of data, eg. on Feb 13, we have a glue job that runs to get the dataset for the week of Feb 5-Feb 11. The dataset is set up to refresh as incremental refresh on Feb 13 with window size of 1 week. So, I found some issues yesterday on Feb 12 and did a full refresh for the dataset. Today, new data from the glue job was generated but quicksight incremental refresh did not ingest any rows.
So in this case, did the full refresh remove the entire refresh layer? Previous data only contained up to the week of Jan 29. How should I prevent this from happening in the future?
Hello @zoomjin, that is interesting, I haven’t experienced this before. The manual full refresh shouldn’t cause an issue with the scheduled incremental refresh. Is it possible that you had already pulled in all of the recent data from the last week in the full refresh on Tuesday so it didn’t have any new rows to pull in on the incremental refresh? Also, when you check the refresh information on the dataset, do you see that the refresh ran?
Maybe to be safe, you could edit the refresh schedule and just set it up for the next coming Tuesday to make sure it is still running as expected.
Yeah, it was a surprise for me to find out the refresh did not happen on Tuesday.
The glue job ran on Tuesday morning, so fresh data did come in, and the full refresh from the day before should not have brought in the data.
The incremental refresh run 2 hours after the job, and I see 0 rows brought in from the incremental refresh. When i did the full refresh again, the total number of rows was higher than the total row count from incremental refresh.
Thank you for the suggestion, yes, I edited the refresh schedule and allow it to set up for next week, also increased window size to 3, not sure if it is going to make a difference though.
Hello @zoomjin, just to make sure, the incremental refresh for this dataset has worked in the past, correct? Also, maybe there was an issue tied to the error you were seeing that caused you to run the full refresh to begin with. Maybe the date field you are using in the database isn’t updated as expected or maybe an issue of how that date field is formatted.
Hello @zoomjin, Thank you for letting us know. I would recommend filing a case with AWS Support where we can dive into the details so that we can help you further. Here are the steps to open a support case. If your company has someone who manages your AWS account, you might not have direct access to AWS Support and will need to raise an internal ticket to your IT team or whomever manages your AWS account. They should be able to open an AWS Support case on your behalf. I hope this helps!
Let me know if you have any further relevant questions, otherwise I will archive this topic for our support team. Thank you!
Hello @zoomjin, since we have not heard back, I will go ahead and archive this topic. If you need further assistance on this or another issue, please post a new question in the community. That will ensure you are at the top of the priority list for a response from one of our QuickSight experts. Thank you!