Hello Community,
can we implement a dependency check in quicksight and have the dataset refresh only if the backend databricks table has latest data in it. if it doesnt have latest data, hold the refresh of datasets and stop sending report snapshot email alerts to users?
Hi @Shivananda, QuickSight has no way to tell if the databricks table has latest data or not, but if you have any other way to detect this outside QuickSight, for example a Lambda function that queries the databricks table on certain interval via EventBridge and checks if it’s updated from the last polling, then you can call CreateIngestion API to dynamically trigger the dataset refresh.
You then call ListIngestions API to check the IngestionStatus for a given IngestionId, and if it is “COMPLETED”, you finally call StartDashboardSnapshotJobSchedule to send email report using one of the configured email report settings in your dashboard.
You can put all these steps together in a workflow using Step Functions. Or, you can use make it event driven - QuickSight recently supported emitting dataset update event to EventBridge so you can simply call StartDashboardSnapshotJobSchedule API upon dataset update event.
1 Like