I created a data set that refreshes daily from a data source that is linked to my S3 bucket in my AWS account. When I refresh that data it shows that more rows have been added during the refresh but once I go back to my analysis page or my dashboard the new data that was added after the refresh does not reflect in my dashboard/analysis at all. I tried everything from making sure the filters are right to deleting and adding the data-source again but nothing seems to work and its stuck on the data that was gathered after I first added that data-set/source into my dashboard.. I checked my S3 bucket to confirm that the data got added and it did so im not sure where is the problem exactly.
Here is how my pipeline works.
Data gets collected in a spreadsheet on SharePoint.
An office script collects all the data and gather it into a separate sheet then downloads that sheet and sends it to the S3 bucket.
Every time the office script runs it sends the updated file to the S3 then a Lamba code deletes the old file and replaces it with the new.
The data is then refreshed in quick and shows how many more rows were added after the new refresh successfully
Problem: The refreshed data does not reflect in the dashboard or the analysis page. (Im using spice)
Hi @Mohamed_Anwer
Based on what you’ve described, this is most likely a dataset reference mismatch. When a data source is deleted and re-added, QuickSight often creates a new dataset with a different internal ID. Your analysis and dashboard may still be pointing to the original dataset rather than the one that’s being refreshed successfully.
Here’s how to verify:
- Navigate to
Datasets in QuickSight and locate the dataset that is refreshing successfully (the one showing the new rows).
- Open that dataset and go to the
Usage tab (please see the screenshot for reference).
- Check which analyses and dashboards are currently associated with this dataset.
If your dashboard/analysis is not listed under the Usage tab, that confirms they are still connected to the old dataset.
Additionally, since you’re using SPICE, your analysis and dashboard will only display the data that has been ingested into SPICE memory for that specific dataset. If they’re referencing a different dataset, the refreshed data won’t appear regardless of how many successful refreshes you run.
Hey @Xclipse
it looks like both the the analysis and the dashboard are listed under usage. How do I make sure that this data is ingested into SPICE memory?
Like I said previously it seems like the data is only showing from the first upload when I first connected my dashboard to the dataset anything after the first refresh is not showing.
Hi @Mohamed_Anwer
Thank you for confirming. It’s good to know the analysis and dashboard are both properly associated with the correct dataset. That helps us narrow things down significantly.
Since they are properly linked, the issue likely relates to how the data is being ingested into SPICE. I’d like to walk you through a few steps to help us identify the root cause
1. Preview the SPICE data directly:
- Go to
Datasets → select your dataset → click Preview data
- This will show you exactly what is currently stored in SPICE memory
- If the new rows appear here, the issue may be at the visual or filter level. If they don’t, it means the refresh isn’t ingesting the updated data as expected.
2. Try a manual Full Refresh:
- Open your dataset → go to the
Refresh tab
- Click
Refresh now and select Full refresh
- A full refresh completely replaces all data stored in SPICE with whatever is currently in your S3 file, rather than appending incrementally
- This is recommended for your setup since your Lambda replaces the entire file each time
3. Verify the S3 connection configuration:
- Open the dataset → click
Edit → check the data source settings
- Confirm that the S3 URI or manifest file points to the exact file key that your Lambda is uploading
- Since your Lambda deletes and replaces the file, please ensure the file name and path remain exactly the same after each upload
For option 1 the Preview data only shows up to 1000 rows nothing more than that. I have about 1400 rows and I can confirm that 1000+ rows are being reflected in my dashboard. (without the updated rows from my S3 of course)
For option 2 I did multiple full refreshes and it did not work
for option 3 yes the file name and path are the same every time there is an upload and I confirmed that the Manifest points to the exact file key.
What worked!!
I had to Basically make a Brand new analysis and add the same data set from the previous analysis and it worked. Only down side is that I had to re-add all my visuals and data to the new analysis and then publish it to replace my current dashboard now the Dashboard updates automatically every time a file is uploaded. For some reason quick just wont update the dashboard and the old analysis when a data sit is switched
Hi @Mohamed_Anwer,
Thank you for the update, and I’m glad to hear you were able to get it working.
What you experienced is a known behavior. When a dataset is deleted and re-added (or replaced), the dashboard still retains a reference to the original dataset. To resolve this, you need to republish the dashboard so it picks up the correct dataset association from the analysis.
For future reference, you can use the describe-dashboard CLI command to get clear details about which analysis and dataset ARNs (DataSetArns) a dashboard is currently using. This can be very helpful for troubleshooting similar issues. describe-dashboard — AWS CLI 2.35.11 Command Reference
To help prevent this from happening again, here are a couple of best practices:
-
Use Replace dataset instead of deleting/re-adding: If you ever need to change the data source in the future, go to your Analysis → click the dataset icon → select Replace dataset. This preserves all your visuals and field mappings while pointing to a new or updated dataset. Replacing datasets - Amazon Quick
-
Avoid deleting the data source when troubleshooting: If refresh issues arise again, try editing the existing data source connection rather than removing and re-creating it, this preserves the internal dataset ID that your analysis and dashboard rely on.
