Evolving list of columns and Glue

Q&A
,
1

My data with 230 columns of data is coming in via files posted to S3. Glue is of course providing the initial schema and I’m having to modify it to meet my needs. My challenge is that as new files come in they often have a column removed or added. If I change my schema to match new files, then the old files don’t line up correctly. Creating a new Glue Crawler and S3 drive each time the columns change is a heavy price to pay so I’m hoping someone has a better solution?

2

This community is focused on AWS Quick Sight.

In regards to quicksight… you can edit the datasets via api calls to conform to the new schemas.