Quicksight spice data refresh

[praveen_suresh]

Hi,
I’m experiencing multiple dataset refreshes with a time interval of 5 minutes. Due to the large amount of data in each dataset, the refresh process takes considerable time. Consequently, the next dataset refresh begins before the previous one completes, creating a loop that extends the overall time required for refreshes. For instance, some datasets may take more than 1 hour to refresh. How can we address this issue?

[apjvinod]

Hi @praveen_suresh

I could think of two approaches …

1. You could try to schedule the incremental SPICE refresh frequency after considering average time taken to complete SPICE refresh . Say average incremental refresh 45 min then you could schedule the incremental refresh to run every 1 hr.

2. If multiple datasets needs to be refreshed along with dependency in a regular interval , then would recommend to orchestrate the SPICE ingestion using AWS Step Functions or Amazon Managed Workflows for Apache Airflow .

Updating Existing Dataset with aws Step Functions (UpdateDataSet)

Hi @gigaform welcome to QuickSight Community !!!. Updatedataset API is to update the definition of the dataset and in order to refresh the data please use Create Ingestion API . And createIngestion API submit jobs in aysnc way , if required please loop through and check for the status of the ingestion using DescribeIngestion API to determine the state of refresh and continue with next steps. Please find the below sample step function to start ingestion and check for its completion. Instead yo…

Thanks
VInod

[praveen_suresh]

Hi Vinod,

Thanks for your help. I think the second solution would work. Let me try that once.

Thanks a lot

Regards
Praveen Kumar