Hi David,
The dataset which is created from a manifest for files in Amazon S3 should have the same number of fields ( columns ) . Amazon S3 Files
In your use-case, the schema is changing. You will have to accommodate these changes as part of your data pipeline.
An example solution below , there will be costs associated if you are testing the solution:
Store those files in S3 . Use an interactive query service like Amazon Athena ( Workshop Studio ) . Within Athena you can create a table that points to location of S3 bucket (S3 , Athena and QuickSight have to be in the same region ) .
Every time there is a schema changed based on the underlying files, you also need to update the table definition in Athena.
Amazon QuickSight can connect to Athena as a data source. Do note that with changing schema, you will also need to refresh your dataset for the new columns to be visible .
Use the newly created dataset in your analysis.
Regards,
Koushik