How do I append to the 1st dataset that was created using csv files ? with a 2nd csv file.
Hi,
You can do it before data load in dataset.
I did not see any option of append multi files in dataset.
There are some other tool like S3 or Athena you can do it.
regards,
Naveed Ali
do you mean that the csv files will be updated in a regular time(e.g. daily)?
for instance, day1.csv has Jan1 data. day2.csv has Jan2 data. Then you want to show all data in QuickSight
If it is the case, you may refer to this approach shared by @darcoli using Athena as data source
Thanks Naveed, can u give me some leads / to try other options
Best regards
Hi,
You can use store your files on S3 using pipelines or Lambda function.
Use Athena to create one table and use this tables in QuickSight.
Regards,
Naveed Ali
Thank you. You have nicely described it. I am not familiar using S3 buckets or Athena hence will need some direction with both S3 and Athena
I will like to give that a try
Best regards
I created S3 bucket and loaded files. Used a note pad to create manifest file and added the reference to the S3 bucket. When to quicksight s3 bucket and ran/ uploaded the manifest file.
This works fine. The only issue I have with larger files is the S3 bucket loading did not complete for file size of 32 GB. The loading process times out after 22 percent loading.
Another way to do this is to simply point the quick site to redshift table.
Thanks
can check if the files have exceeded the limit? Data source quotas - Amazon QuickSight
As the file size are quite big(32GB), suggest to use incremental load if it is possible to reduce data loading time. If the data size is huge or there is already a Redshift in place, suggest to use it.
Hi @rajabhau. Did Roy’s feedback help? Hope so. Let us know if this is now solved. (Or, please mark the Solution.) Thanks!