Hi all,
I run an ETL which stores a .csv file in s3. From there, I import it to QS as I have created a dataset linked to that s3 uri. Now, I changed the ETL and the .csv has slightly changed since I removed two fields from the dataset. When I now try to refresh it fails…Anyone know why? Do every time I need to apply changes to my dataset I will have to create a new one??
Hello @Fotis_flex, when you say 2 fields were removed from the dataset, do you mean that 2 columns have been taken out? Since it is a CSV that is being imported the structure needs to be maintained in order to update it with new data. You should be able to add rows and new columns and update it without an issue, but I believe any time you remove columns it will need to be set as a new dataset or it will error. This is the error that displays when trying to update a current CSV dataset with one that eliminited 2 columns:
Hello,
As @DylanM mentioned, for this case, it is needed for you to keep the fields and the order “Your new file needs the same fields in the same order as the original file. New fields can be added after existing fields.”
If you do not want to recreate the dataset you can just leave the columns that you want to remove empty in the ETL and exclude them in data preparation so they are not used in any visual/analysis.
Here is a link to our documentation with some recommendations about Changing datasets.
Hope this helps!
1 Like
@andres007 and @DylanM Thanks for info. Yes, that’s my conclusion too. I don’t find the solution very attractive but I suppose the best thing to do is exclude them. I tried updating the “dataset description” from cli where you can define the dataset w/o the columns. It was successful but it messed up my analysis (which didn’t include the fields). Anyway, @andres007 if I simply exclude the fields and share the dataset with other users (viewers) will they be able to include them again or the fields will be completely invisible to them?
If you exclude them in the Dataset, they will not be able to include them again if they do not have access to modify the Dataset.
1 Like