Latest data is not reflecting in the Dashboards after Schedule Refresh on Daily basis at 2 PM IST

Question & Answer
, , , , , ,
1

Hi, I have created a dashboard using 5 individual files ( my team will upload these CSV files in 10 AM IST ) placing in S3 and i set up Daily Schedule refresh at 2 PM IST to reflect latest data in the Dashboards. But unfortunately this not happening for the since the beginning of creation of the dashboard as i was getting every second messages from the Customers/Leadership why/what this is happening with Quicksight.

I am getting mad at this QS behaviour?

What’s the issue with QS. Anyone from QS team ?

2

Hi @Venkat.Metadata

I have a few questions for you:

  1. What is the data source of your five individual files? If it’s S3, how are you connecting them to QuickSight — are you using manifest files?
  2. Is there any ETL process involved? For example, does any transformation or trigger occur when your team uploads the files?
  3. Are there any schema changes during the uploads — such as additional columns or modifications to the structure?
  4. How is the dashboard configured to reflect the latest data? Are you using any date field or filter to ensure freshness?
  5. Have you reviewed the Refresh History in QuickSight? Are there any errors or failures reported?

If possible, please add some screenshots of refresh schedule etc.

Thank you,
Shravya

3

Hi @shravya

  1. 5 individual csv files placed in S3 ( manifest files are there too ). I created 5 datasets after giving the path of Manifest file ( Manifest files contains path of respective CSV file path )

  3. They are uploading 5 individual files directly replacing ( or deleting ) the older files. There is no schema changes… structure, column names , data types are same in every file. no additional columns adding, ( i will get email when refresh fails ).

  4. I have set up schedule refresh to the datasets ( Is there Schedule refresh to the Dashboards separately ? )

  5. I don’t see error in the refresh process.

  6. At 2 PM, refresh will takes 3-5 minutes , over 1000 rows only

when I open the dashboards after refreshing time, i don’t see the latest data. ( Latest data is there in CSV files in S3 )

But when I open the each individual datasets and after doing save and publish then only i can see the latest data when i open dashboards after doing the datasets first.

1 Like
4

@Venkat.Metadata - Since you mentioned that manual refresh works, it likely re-reads from S3 and fetches the latest content.

I believe when you use individual files via a manifest file, QuickSight doesn’t update unless there are changes in manifest file or paths etc. I know it will be frustrating manually refreshing the data everyday.

I’d recommend setting up an AWS Glue Crawler to crawl the S3 bucket on a daily schedule (e.g., 10:30 AM IST since you said your team uploads at 10 AM). This will ensure your Athena tables are up-to-date. Once linked to QuickSight, the dataset refresh should reflect the latest changes automatically.

Thank you,
Shravya

5

Hi @Venkat.Metadata,
It’s been awhile since we last heard from you on this thread, did you have any additional questions regarding your initial post?

If we do not hear back within the next 3 business days, I’ll close out this topic.

Thank you