Appending dataset using csv files - data source is multiple csv files with same column names

[rajabhau]

How do I append to the 1st dataset that was created using csv files ? with a 2nd csv file.

[Naveed]

Hi,
You can do it before data load in dataset.
I did not see any option of append multi files in dataset.
There are some other tool like S3 or Athena you can do it.

regards,
Naveed Ali

[royyung]

do you mean that the csv files will be updated in a regular time(e.g. daily)?
for instance, day1.csv has Jan1 data. day2.csv has Jan2 data. Then you want to show all data in QuickSight

If it is the case, you may refer to this approach shared by @darcoli using Athena as data source

Append new .CSV file data?

In the meantime, another simple approach might be to store these csv files in an s3 bucket and add an Athena table reading these files. Then, from QuickSight, use that athena table in your dataset rather than uploading the files there. (Assuming you don’t use Athena partitioning, then newly updated files’ data will immediately be visible to QuickSight - but will need a SPICE refresh if you are using that)

[rajabhau]

Thanks Naveed, can u give me some leads / to try other options

Best regards

[Naveed]

@royyung

using Athena as data source

Hi,

You can use store your files on S3 using pipelines or Lambda function.
Use Athena to create one table and use this tables in QuickSight.

Creating tables in Athena - Amazon Athena

Regards,
Naveed Ali

[Naveed]

Hi,

This may help you.

How to use SQL to Query S3 files with AWS Athena | Step by Step Tutorial

[rajabhau]

Thank you. You have nicely described it. I am not familiar using S3 buckets or Athena hence will need some direction with both S3 and Athena

I will like to give that a try

Best regards

[rajabhau]

I created S3 bucket and loaded files. Used a note pad to create manifest file and added the reference to the S3 bucket. When to quicksight s3 bucket and ran/ uploaded the manifest file.

This works fine. The only issue I have with larger files is the S3 bucket loading did not complete for file size of 32 GB. The loading process times out after 22 percent loading.

Another way to do this is to simply point the quick site to redshift table.

Thanks

[royyung]

can check if the files have exceeded the limit? Data source quotas - Amazon QuickSight

As the file size are quite big(32GB), suggest to use incremental load if it is possible to reduce data loading time. If the data size is huge or there is already a Redshift in place, suggest to use it.

[Kristin]

Hi @rajabhau. Did Roy’s feedback help? Hope so. Let us know if this is now solved. (Or, please mark the Solution.) Thanks!