Most of our users ask us for best practices for CI/CD (a.k.a. BIOps) in QuickSight to plan, control, validate and finally deploy changes in their QuickSight environment.
While this is a pretty broad topic that we recommend to discuss with your AWS Solution Architect we are providing here some general recommendations and details of the different tools that both, QuickSight and the AWS ecosystem provides for you to accomplish this task.
As a summary we will be covering the following:
- QuickSight APIs (Templating capabilities and also defining assets as code)
- Cloudformation/CDK (infrastructure as code)
- (optional) Use of AWS Code services
Before continuing, a couple of recommendations to implement these best practices:
- Consider using separate AWS (and hence QS) accounts for the different environments (Dev/Test/Pre-production/Production), this is in line with the AWS best practices whitepaper so you can, amongst other benefits, reduce the blast radius in case of miss-configurations or accidental code/setup changes in an account.
- Use a process to automatically promote changes from development to Test accounts for example but include manual approval processes when promoting to pre-production and prod.
- Use different datasets for development and testing than the ones you will have in pre-production and prod (to prevent access to PII and customer data in early development phases)
Now, with these recommendations in place, lets dive deep on the aforementioned items one by one:
QuickSight APIs
QuickSight provides a wide set of API operations for you to automate repeatable tasks (such as continuous deployment). In this case QuickSight offers two options for you to replicate
- Templates (allows you to keep de design of your dashboards and analyses) with the option to replace the datasets that feed them. Allowing you to easily replicate exact copies of your assets between accounts and regions replacing the underlying datasets (allows you to meet best practices 1. and 3.)
- Expanded APIs capabilities Allows you to completely export the design of your dashboards and analyses in text format (JSON) so you can version control in your own CVS and also make modifications (by editing the JSON) as needed before re-deploying the assets in the target accounts and region(s)
As you can see these are two options that allow you to replicate assets between accounts (templates will make exact copies of the assets) while the expanded API capabilities will allow you download the asset description, store it and also modify it as needed.
In terms of whether to use one or the another it depends on your use case, for a CI/CD where you don’t need to store assets in your own CVS and you just need to create a CI/CD Templates are simpler and enough (they also support versioning) the only restriction is that you cannot export them outside of QuickSight to upload them to your CVS for example.
Cloudformation and CDK
Cloudformation and the Cloud Development Kit (CDK) are two options to define infrastructure as code in AWS. Cloudformation uses a JSON or YAML notation to build resources in AWS while the CDK provides a high level abstraction in several programming languages to define assets in Cloudformation.
Bot options, Cloudformation and the CDK support creating QuickSight assets (Datasources, Datasets, Analyses, Templates and more). The expanded API capabilities are not yet supported but will be announced soon.
Cloudformation or the CDK handles the orchestration of the required API calls to create assets for you, so you don’t need for example, to issue the delete operation of an asset (e.g dashboard) to have it updated, if Cloudformation detects that in order to update an existing asset a delete and replace is needed it will do it automatically for you.
Other Cloudformation benefits are the option to use stack sets that allows you to deploy to multiple accounts and regions at a time, the detection of drifts in assets configurations (so yo can see if an asset has been modified manually, e.g via the QS console) and more …
AWS Code services
Last but not least, with the above pieces (low level APIs + orchestration and asset definition) we are now ready to use the AWS code services such as Code Build or Code Pipeline.
Code pipeline will allow us to define deployment automation with the option to create test/approval intermediate stages to limit the automatic promotion to critical environments such as pre-production or production (meets recommendation 2.) without slowing down innovation from development to test (with a full CI/CD).
Code Deploy will be used in the deployment stages in conjunction with Cloudformation to deploy the QuickSight assets in the different environments.
The use of these services is optional as you can also rely on other CI/CD solutions such as Jenkins, Bamboo etc …
Being conscious that there is a big number of pieces in this architecture I have created a simple high level diagram of how it would look like for your reference:
Hope you find this article interesting!
Happy dashboarding!