Today I encountered a very strange issue. I use an ETL pipeline that creates .csv files in S3. I have a QS dataset that is connected with one of the .csv files in S3 out of which it’s being refreshed. I just added a new field to this .csv and pressed “Refresh” on QS side so that I can update the dataset (as I always do). Funny thing is that although the dataset is being successfully refreshed, it keeps its old schema and the field does not appear anywhere. If I go “edit dataset” and click on dataset preview, the field appears as expected. What might be wrong here? My field name is called tags and you see the pics below:
Based on the information and screenshots you provided, it appears that the issue you’re facing is related to the way QuickSight handles schema updates when refreshing datasets.
QuickSight has a feature called “Schema Persistence,” which is designed to maintain the schema of a dataset even when the underlying data source changes. This feature can be useful in scenarios where you want to preserve the existing schema and avoid breaking any existing visualizations or analyses that rely on the current dataset structure.
However, in your case, where you’ve added a new field (tags) to the CSV file, QuickSight is not automatically detecting and incorporating the schema change during the dataset refresh.
Here are a few potential solutions or workarounds you could try:
Manually Update the Dataset Schema:
Go to the QuickSight dataset editor.
Click on the "Edit" button next to the dataset schema.
In the schema editor, you should be able to add the new "tags" field manually.
Save the updated schema.
Refresh the dataset again.
Disable Schema Persistence:
Go to the QuickSight dataset editor.
Click on the "Data Source" tab.
Locate the "Schema Persistence" option and disable it.
Save the changes.
Refresh the dataset again.
With schema persistence disabled, QuickSight should detect and incorporate the new field from the CSV file during the refresh.
Create a New Dataset:
If the above solutions don't work, you can try creating a new dataset from the updated CSV file. This will ensure that QuickSight detects the new schema correctly.
Check for Naming Conventions:
Ensure that the new field name "tags" follows QuickSight's naming conventions for field names. QuickSight has certain restrictions on field names, such as not allowing spaces or special characters.
Check for Data Type Compatibility:
Verify that the data type of the new "tags" field is compatible with QuickSight's supported data types. If the data type is not recognized, QuickSight may ignore the field during the refresh.
If none of these solutions work, you may need to investigate further or reach out to AWS Support for assistance, as there could be an underlying issue preventing QuickSight from detecting the schema change correctly.
@murili thanks for reply but I can nowhere find the way to “Manually update the dataset schema” or “Disable schema persistence”! I simply can’t see any of the above options when I click “Edit dataset” I have been adding fields like this for quite some time now and every time the field was appearing in the edit preview so when I clicked on “Save and Publish” everything was going fine. The option to create another dataset is not really solving anything as this dataset is also shared with customers that have created their ownn analysis with it. Can you please paste some screenshots of the steps of the first 2 options? Thanks
Hello @Fotis_flex, were you able to resolve this issue that you are facing? If not, I am curious, is the base dataset the one you are bringing in from S3 and then you are joining the CSV on that dataset? Also, is there any chance you are hitting a dataset limitation? Do you already have 2,000 columns or are you at one of these dataset size limits?
If you are still looking for a solution here, I can try to guide you with a little more information. Thank you!
Hello @Fotis_flex, since we have not heard back from you with any follow-up information, I will close out this topic. If you still need assistance, please post a new question in the community and include a link to this topic to provide relevant information. That will ensure you are at the top of the priority list for a response from one of our QuickSight experts. Thank you!
