I wanted to follow up on the case we discussed last October. I’m referring to this thread: here
I wanted to ask whether there have been any updates regarding this feature request.
We are still experiencing significant challenges with schema changes during ongoing dataset development and maintenance. Since our datasets evolve continuously, functionality in this area would save a considerable amount of time and help reduce disruptions that can impact our customers’ dashboards.
We consider this issue quite critical for our use case, so I wanted to reach out and see whether there has been any progress or any planned improvements.
Thank you very much for your time and support! It is much appreciated. I look forward to hearing from you.
Hi @Fotis_flex,
Could you provide some more detailed information on the challenges/issues you’re experiencing when making schema changes?
To my understanding, AWS has been working on adding additional aspects and feature inclusions around the new data prep experience; if you could provide an example of a scenario that causes issues, we may be able to assist further with recommendations.
Hi @Fotis_flex,
I just came across this Technical Article that was recently published as well that I believe is somewhat relevant to your scenario. Let me know if this helps provide some additional insight!
Thank you for your reply and for sharing the documentation.
While it is very close to what we’re looking for, I don’t think it addresses our use case. The documentation focuses on QuickSight datasets that point directly to database tables and explains how to handle schema changes in that scenario.
In our case, however, QuickSight ingests data from S3 files, not directly from Redshift or another database. These S3 files are updated regularly by our ETL pipeline.
The challenge is that whenever we add or remove fields, rename columns, update descriptions, or make any other schema changes, we have to manually update the QuickSight dataset schema. This creates operational overhead and delays.
For example, imagine an ETL pipeline that runs every 8 hours and unloads the latest data to S3, which serves as the QuickSight data source. If the pipeline is enhanced to include new fields, the next scheduled QuickSight refresh fails because the dataset schema still reflects the previous version. As a result, someone has to manually update the schema before the refresh can succeed, delaying the availability of the new data to our customers.
Ideally, we would like the ability to prepare or stage schema changes in advance so that QuickSight uses the updated schema automatically during the next scheduled refresh. At the same time, the currently active schema should remain unchanged until that refresh occurs, ensuring that existing dashboards continue to work without interruption.
Is there any way to achieve this kind of workflow in QuickSight?
Hi @Fotis_flex,
Thank you for clarifying a bit further in detail, unfortunately this is not a possibility quite yet. I will leave marked as a feature request to promote visibility to the support team.