Automate your Amazon QuickSight assets deployment using the new Amazon EventBridge integration

amazonquicksight · December 8, 2023, 9:39pm

Business intelligence (BI) and IT operations (BIOps) teams often have the need to automate and accelerate the deployment of BI assets to make sure of business continuity. We heard that you wanted an automated, scalable way to deploy, back up, or replicate Amazon QuickSight assets at scale so that BIOps teams within your organization can work in an agile manner.

We are launching QuickSight assets Create, Update, and Delete (CUD) events using Amazon EventBridge. With EventBridge, developers can respond automatically to events in QuickSight such as a new dashboard creation or update. These events are delivered to EventBridge in near real time. Developers can write simple rules to indicate which events are of interest to them and what actions to take when an event matches a rule. By subscribing and responding to QuickSight events in EventBridge, customers can automate their workflows, such as continuous deployment, replication, and backups. This launch covers Create, Update, and Delete events for

Dashboards
Shared folders
VPC connections
Analyses

Feature overview

All events payloads have account ID, Region, and TimeStamp in UTC. The following table shows the events with event specific details.

Asset type	Event	Common details	Event specifc details
Dashboard	QuickSight dashboard creation successful	DashboardId, VersionNumber	.
	QuickSight dashboard creation failed		Errors
	QuickSight dashboard update successful		.
	QuickSight dashboard update failed		Errors
	QuickSight dashboard published version updated		.
	QuickSight dashboard deleted		DashboardId
Analysis	QuickSight analysis creation successful	AnalysisID	.
	QuickSight analysis creation failed		Errors
	QuickSight analysis deleted		.
Folders	QuickSight folder created	FolderId	ParentFolderArn (If applicable)
	QuickSight folder updated		.
	QuickSight folder deleted		.
	QuickSight folder membership updated		MembersAdded[], MembersRemoved[]
VPC Connection	QuickSight VPC connection creation successful	VPCConnectionId, AvailabilityStatus	.
	QuickSight VPC connection creation failed		.
	QuickSight VPC connection update successful		.
	QuickSight VPC connection update failed		.
	QuickSight VPC connection deletion successful		.
	QuickSight VPC connection deletion failed		.

Use case

Let’s consider a fictional company, AnyCompany, that owns healthcare facilities across the globe. They have set up a development QuickSight account for authors to create and update QuickSight assets and a separate production account. Additionally, in some cases, due to data residency regulation, they have to maintain the same assets across multiple Regions. AnyCompany is scaling its business and they want to automate deployment within and across multiple QuickSight accounts and back up QuickSight assets as soon as there are changes in an asset lifecycle.

AnyCompany has the following key deployment and backup requirements:

Continuous Deployment: AnyCompany wants to automate the deployment of QuickSight assets from their development to their production account. The production accounts for some teams are in different Regions whereas for others they’re within the same Region. AnyCompany wants to automate this process of deployment. AnyCompany uses QuickSight shared folders for the DevOps team to pick up any updated assets. When an author is done with their changes, they update the dashboard in a specific shared folder. Until now, the author had to inform the DevOps team that the dashboard was ready for deployment, which is manual and not efficient.
Automated Backup: As AnyCompany rolls out critical dashboards for business, it needs to make sure that dashboards are highly available . As part of their strategy, AnyCompany wants to maintain a backup of assets to restore in case of disasters. Until now, AnyCompany has been backing up assets on a fixed schedule. They want to reduce the delay between a change in an asset and backup.

Continuous deployment

When an author is done with their changes, they update the dashboard in a specific shared folder. In case the dashboard is no longer needed in production, they simply remove the dashboard from the folder. DevOps has created a rule in EventBridge to capture and send folder membership update events to a Lambda function, which then uses QuickSight export and import APIs to deploy the updated dashboard from the development account to the production account.

DevOps chooses the following event and creates a rule to send it to a Lambda function

QuickSight folder membership updated

{
  "version": "0",
  "id": "3acb26c8-397c-4c89-a80a-ce672a864c55",
  "detail-type": "QuickSight Folder Membership Updated",
  "source": "aws.quicksight",
  "account": "123456789012",
  "time": "2023-10-30T22:06:31Z",
  "region": "us-east-1",
  "resources": ["arn:aws:quicksight:us-east-1:123456789012:folder/77e307e8-b41b-472a-90e8-fe3f471537be"],
  "detail": {
    "folderId": "77e307e8-b41b-472a-90e8-fe3f471537be",
    "membersAdded": ["arn:aws:quicksight:us-east-1:123456789012:analysis/e5f37119-e24c-4874-901a-af9032b729b5"],
    "membersRemoved": []
  }
}

Creating the EventBridge rule

Open the Amazon EventBridge console and in the navigation pane, choose Rules.
Choose Create rule. Enter a name and description for the rule. For example, enter QuickSightAssetChangeRule.
Select default Event bus.
Choose Rule with an event pattern, and then choose Next.
For Event source, choose AWS events or EventBridge partner events.
Scroll down to the Creation method section.
1. Choose Custom pattern (JSON editor).
2. In the Event pattern text box, enter the following code snippet and choose Next.

{
  "source": ["aws.quicksight"],
  "detail-type": [{
    "anything-but": "AWS Service Event via CloudTrail"
  }]
}

For Target types, choose AWS service and Lambda function.
For Function, choose the Lambda function that you created and then choose Next.
In Configure tags, choose Next and then choose Create rule.

Once the Lambda function gets invoked, it uses the following QuickSight APIs to export the updated dashboard from the development account and import it into the production account. Details of how to use these APIs can be found in the QuickSight Developer Guide and Automate and accelerate your Amazon QuickSight asset deployments using the new APIs.

Export APIs

You can use the following APIs to initiate, track, and describe the export jobs that produce the bundle files from the source account. A bundle file is a zip file (with the .qs extension) that contains assets specified by the caller and, optionally, all dependencies of the assets.

StartAssetBundleExportJob – Use this asynchronous API to export an asset bundle file.
DescribeAssetBundleExportJob – Use this synchronous API to get the status of your export job. When successful, this API call response will have a presigned URL to fetch the asset bundle.
ListAssetBundleExportJobs – Use this synchronous API to list past export jobs. The list will contain both finished and running jobs from the past 15 days.

Import APIs

These APIs initiate, track, and describe the import jobs that take the bundle file as input and create or update assets in the destination account.

StartAssetBundleImportJob – Use this asynchronous API to start an import of an asset bundle file.
DescribeAssetBundleImportJob – Use this synchronous API to get the status of your import job.
ListAssetBundleImportJobs – Use this synchronous API to list past import jobs. The list will contain both finished and running jobs from the past 15 days.

def lambda_handler(event, context):
    try:
        logger.info('Start')
        logger.debug(event)
        
        #Extract folder id from event
        folderId = event['detail']['folderId']
        eventTime = datetime.datetime.strptime(event['time'], '%Y-%m-%dT%H:%M:%S%z')
        logger.info('Folder id  - ' + folderId)
        logger.info('Event time - ' + str(eventTime))
        
        
        
        #Get the list of all members in our folder.
        listFolderMembersResponse = quicksightSource.list_folder_members(
                AwsAccountId = awsAccountId,
                FolderId = folderId
            )
        logger.debug(listFolderMembersResponse)
        
        #Extract folder member arns
        folderMemberArns = [ member['MemberArn'] for member in listFolderMembersResponse['FolderMemberList']]
        logger.debug(folderMemberArns)
        
        
        if len(folderMemberArns) == 0:
            #Nothing to process if folder is empty
            logger.info('Folder is empty. Not processing further.')
            return
        
        #Multiple events will be fired when you add multiple assets to the folder.
        #In this sample, we are extracting all folder members regardless of which member triggered this lambda.
        #You can focus the logic on the triggering member if that is better suited for your use case.
        #Since we are operating at folder level, checking if another export is running.
        WaitForExportToComplete(folderId, eventTime)
        
        logger.info('Starting export job')
        logger.info(str(datetime.datetime.now(datetime.timezone.utc)))
        
        #Start asset bundle export
        startAssetBundleExportJobResponse = quicksightSource.start_asset_bundle_export_job(
                AwsAccountId = awsAccountId,
                AssetBundleExportJobId = 'FolderExport-'+folderId,
                ResourceArns = folderMemberArns,
                ExportFormat='QUICKSIGHT_JSON',
                IncludeAllDependencies=True
            )
        
        #Wait for the above invoked export job to complete. Passing current time to bypass the time check.
        WaitForExportToComplete(folderId, datetime.datetime.now(datetime.timezone.utc))
        
        #Download the asset bundle file
        DownloadExportBundle(folderId)
        
        #Read contents of the bundle file
        fileHandle = open('/tmp/FolderExport-'+folderId+'.qs', mode="rb")
        fileContent = fileHandle.read()
        
        #Start asset bundle import in target region
        assetBundleImportJobresponse = quicksightTarget.start_asset_bundle_import_job(
                AwsAccountId = awsAccountId,
                AssetBundleImportJobId = 'FolderImport-'+folderId,
                AssetBundleImportSource={
                    'Body': fileContent
                }
            )
        logger.debug(assetBundleImportJobresponse)
        
        #Wait for above import job to finish
        WaitForImportToComplete(folderId)
    
    except Exception as e:
        if str(e) != 'NO_FURTHER_PROCESSING_NEEDED':
            logger.error(str(e))

Automated backup

AnyCompany deploys business-critical dashboards, and it’s important for them to have proper backup and version control processes. In addition to running scheduled export jobs at regular intervals, they also want to back up critical assets as soon as they’re updated. Along with deployment, they back up the exported files. DevOps at AnyCompany has created EventBirdge rules to capture dashboard update events, and when the event gets triggered, they use the assets exported API StartAssetBundleExportJob to export and back up the dashboard in their preferred storage system. They create the EventBridge rule using the same steps as mentioned before and use the following event pattern. This rule will only be triggered when the dashboard with the specified ID is updated.

{
  "source": ["aws.quicksight"],
  "detail-type": ["QuickSight Dashboard Update Successful"],
  "detail": {
    "dashbaordId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  }
}

Conclusion

QuickSight asset events using EventBridge eliminate manual processes or polling for updates and provide a way to automate deployment processes across multiple environments. This post illustrates use cases where you can apply these events for automation. For more information, refer to Amazon QuickSight and the QuickSight Developer Guide.

If you have any questions or feedback, please leave a comment. For additional discussions and help getting answers to your questions, check out the QuickSight Community.

Join the Quicksight Community to ask questions, share answers, learn with others, and explore additional resources.

About the Authors

Mayank Agarwal is a product manager for Amazon QuickSight, AWS’ cloud-native, fully managed BI service. He focuses on embedded analytics and developer experience. He started his career as an embedded software engineer developing handheld devices. Prior to QuickSight he was leading engineering teams at Credence ID, developing custom mobile embedded device and web solutions using AWS services that make biometric enrollment and identification fast, intuitive, and cost-effective for Government sector, healthcare and transaction security applications.

Arun Santhosh is a Senior Solution Architect for Amazon QuickSight. Arun started his career at IBM as a developer and progressed on to be an Application Architect. Later, he worked as a Technical Architect at Cognizant. Business Intelligence has been his core focus in these prior roles as well.

This is a companion discussion topic for the original entry at https://aws.amazon.com/blogs/business-intelligence/automate-your-amazon-quicksight-assets-deployment-using-the-new-amazon-eventbridge-integration/