I have a dataset in Quicksight which has the files from S3(dropped in s3) for the months after May. The dataset has the June files, I dropped a latest file in S3. on **july 11,2023 ** . I want to create a lambda which on execution with automatically refresh the quicksight dataset and show the july 11 file, could someone help in providing the steps and lambda function and ways to do this.
Hi @dsahu,
There are multiple ways to deal with this:
QuickSight supports scheduled refreshes of data from your source. This can be configured directly from QuickSight console from dataset details page. This is the recommended approach as this would not involve any code.
The second option is to use our CreateIngestion API and refresh the dataset.
Thanks,
Sri
1 Like
Hi @dsahu - welcome to AWS Quicksight community and thanks for posting the question. This is an event driven process, when a new file will put in S3, you need configure a lambda function and essentially the lambda function will call boto3 api to refresh the data set ( create ingestion process). See one sample example below - Invoke AWS Lambda Asynchronously Using Amazon S3 to Trigger Lambda | by Mehmet Ozkaya | AWS Lambda & Serverless — Developer Guide with Hands-on Labs | Medium
The quicksight boto3 api details can be found -QuickSight - Boto3 1.28.2 documentation
Also please refer @Srikanth_Baheti suggestion as well.
Regards - Sanjeeb
2 Likes
Thank you @Srikanth_Baheti and @Sanjeeb2022 for your responses. Really appreciate it!
I understand, we can update the dashboard with latest data coming from the source dataset through the console. But I wanted to check, if there is a way, from your other options that you provided, we can see the latest file information in the source dataset itself. For example: if you look into the below screenshot-
the query is written to fetch the files >2023-06-01, I want to see the File, in this dataset which I dropped today, is there a way to see this, so that automatically the dashboard can view the latest information once we refresh it
Not sure if I follow you 100%. but if you are looking to get the SPICE ingestion history then we have an API for that. But it is not going to show you the files in the dataset itself. It will show you when the ingestion was run. ListIngestions API
Not sure the dataset screen you shared is a dataset that you have created separately.
1 Like