I have a dataset that contains only one table imported from aws athena, the table is the result from a Glue ETL job and it is listed on the aws glue data catalogue, in parquet format in s3.
On the other hand, I am creating a state machine in step function that will update the ETL job, and then refresh the dataset in QS so that we always have the most up-to-date dashboard. I can see there is a Amazon Quicksight API called UpdateDataSet and it seems to be what I need. However, I am not sure what one of the parameters, PhysicalTableMap, does?
It appears that it is asking for the column headings and data types in JSON format? Is there a way we can get this easily for tables in aws ?
Essentially, I just want to find a way for step function to press the “Refresh Now” button for me automatically.
Hi @gigaform
welcome to QuickSight Community !!!.
Updatedataset API is to update the definition of the dataset and in order to refresh the data please use Create Ingestion API .
And createIngestion API submit jobs in aysnc way , if required please loop through and check for the status of the ingestion using DescribeIngestion API to determine the state of refresh and continue with next steps. Please find the below sample step function to start ingestion and check for its completion. Instead you can start ingestion using Stepfunction and check the status of refresh in QuickSight Dataset console whenever required.
Thanks
VInod
4 Likes
Thanks it worked, just one quick questions if that is fine, in your screenshot, you have added additional logic to check the state of refresh, which is fine but i have multiple datasets in QS that I need to update, is it possible to just check the wait for callback checkbox so that it will only carry on next steps if the QS dataset refresh is a success?
Hi @gigaform
Thanks for confirming . And “wait for callback” feature will not check for ingestion status completion rather it will pause until another application/step sends success token .
Thanks
VInod
1 Like
