S3 datasets refresh every 5 mins or less than 5 mins

Question & Answer
,
1

Hey Team,

I am looking the answer for the below query:

There is one script that running every 5 mins on ec2 instance and its generate one csv file and that csv file we are uploading into the s3 via the awscli command.
Once file uploaded into the s3 we need to show the content into the quicksight. So we created one s3 datasets and add the manifest file with the url for csv file. So while i go through the options for refresh the datasets i found 3 oprions Manual, Scheduled, and Incremental.
In manual refresh we can do only 32 times per day
In Scheduled refresh we can do min 1 hour
And In Incremental we can do at least 15 mins so how i can acheive the datasets refresh every 5 mins.

If we need to add any services in middle of s3 and quicksights or something else please suggest me

Note - There is no concern about the costing.

Thanks
Nishant Jain

2

Hi @Nishant
At least two ideas
First: you could try to use

Second: you could use Athena as layer between S3 and QuickSight.
BR

3

Hi @Nishant

In addition to what @ErikG mentioned .

Manual refresh of 32 time per day is inclusive of both options QuickSight UI : Refresh Now + QuickSight API Call : CreateIngestion.

Incremental refresh on S3 datasource is not supported since we cannot query data from S3 with filter conditions on the date field . Its better to have a table defined in Athena on top of the S3 bucket and use Athena datasource connector to schedule the incremental refresh for every 15 mins .

If its critical requirement to refresh every 5 mins on incremental mode using create ingestion API , would suggest to talk to your AWS Account team & QuickSight Solution Architect to discuss on the usecase to evaluate the options to increase refresh limits.

Thanks
Vinod