I came across a strange error while working with the QuickSight API (both boto3 and aws CLI). It looks like when a filter is applied to all applicable visuals, the JSON returned by the describe_xyz method contains items with duplicate IDs in the Definition.FilterGroups list.
I’ve tried creating a template from the JSON downloaded using describe_template, but the validation failed because of duplicate IDs. The only difference I can notice between two FilterGroup items with the same ID is in the ScopeConfiguration.SelectedSheets.SheetVisualScopingConfigurations list which contains a single element, but the SheetID is different for each. It looks as if there should really be only one FilterGroup with multiple items in scope.
Hi @Adrian_Vacaru - This is my understanding of what you’re trying to do. You want to migrate some assets (visuals) from one AWS account to another.
You have used the new assets APIs (boto3 and CLI) to generate the JSON scaffolding for those assets but you’re running into an issue, where the JSON generated for a visual doesn’t seem to pass the validation checks when used in the template. The error returned is related to “Duplicate IDs”.
Can I confirm that you have seen and followed the instruction in this blogpost on how to use the new APIs?
Yes your understanding is correct, I’m using the new assets API to download the assets as JSON, particularly describe_template_definition and create_template for this example.
The exact error looks like this:
botocore.errorfactory.InvalidParameterValueException:
An error occurred (InvalidParameterValueException) when calling the CreateTemplate operation:
Duplicate Filter Groups found :
[
7e9f5961-3747-45c2-a010-aa9e1a792fed,
6feb2faa-40ce-47c2-801a-f04faf6bfae2,
a984ed7d-1366-49dc-ab90-5e81dfaa720b,
7e37a0f6-f1ac-4d89-9a45-25faf7abdcd7,
628c1d0a-7d10-41a1-a377-19f667acc71c,
6b2ae905-660e-4d14-8ee7-35d8f081070c,
63f475b9-a039-4659-8f06-699f54c0ace7,
104e4da7-965b-46e7-96a2-37257c1bde3c,
0e6c588b-dbc7-4cb4-ad1e-ed035dbc4f38,
fbac3062-d5a6-48f8-9646-1f9d4f385380,
a8719904-e9b2-4e9e-a309-aaee2db07d21,
499497f9-cb90-4eef-b0fe-721c6901192e,
19dbbc8b-8ce6-4333-aa06-ec3e2ff93735,
b7027212-720c-4be9-94a1-428a24ec9ad0,
78a33677-0829-4ea9-86ec-4ca1cf67a248,
1f5716e0-923c-4d54-95d6-96a30b686ca8,
79cb61bc-141f-4694-aa95-d99f78918402,
5645d164-ab68-4837-8f0a-275648947e89,
5a0885f3-f2bb-4aaa-b80a-ff078af47d99,
5a8766ae-468f-441a-98f9-f2146377159d,
40cf7414-fa24-468b-80e5-661c85d0c8f9,
47b19baa-6fee-4a72-8e3b-f9d8e5dff338,
549d7187-375d-444e-a115-1460c3f46bff,
1b00191f-df68-450b-8a11-97665d2ce9d7,
e4dbcccc-21a3-4a68-a4b9-56bd2c927a56,
b689bbc8-1d97-4541-81ca-ae72e24faf3a
]
I haven’t seen the blogpost, but I had a look at it now, seems very similar to what I’m doing in python.
I have found that the order in which those APIs must be used for you to achieve what you want is a bit tricky. What is the order of the API calls right now?
Hi @eperts , still working on it, currently I’ve managed to export the assets, but I’m trying to figure out a way to import them in the other accounts with the correct override parameters.
I’ll circle back once I get it over the line, but until now these new APIs are promising, I’m glad you mentioned them, thank you!
Hi @Adrian_Vacaru@eperts - Thank you very much. One request , is it possible we can document steps so that this will help whole community. Since it is fairly new API, a good documentation will help us. Hi @Adrian_Vacaru - if your POC is successful, can you please share the details so that I can give a try or if you can document this, we can request QuickSight team to put in a blog post in QuickSight community.
@eperts one thing I feel was missing from the blog post is more details around importing the job and overriding parameters. Maybe some more examples? I’m having some issues figuring out what to override in such a way that my VPC connection and data source still work. Figuring out subnet IDs, security group IDs, credentials, etc.
@Sanjeeb2022 - I will reply here once I get it done and mention some of the use-cases I have (importing in the same account, but different VPC, importing in different account, but multiple VPCs, etc) and how I managed to solve it. It won’t cover all the possible options, but maybe it will help someone.
This is pretty straight forward, start an asset bundle export job with the dashboard ARN and you’ll get a ZIP file with all the resources as directories:
vpcConnection
datasource
dataset
theme
dashboard
Importing an assets bundle
The tricky part comes when you’re trying to import an assets bundle. You can import it to a different AWS account or the same AWS account, but in each case there are a few things to be aware of (especially if you want to automate this):
if you have a label such as dev/staging/prod in your resource IDs or names you’ll have to override this in the import job.
the PrefixForAllResources override configuration is kind of useless if your resource ID contains dev for example. If you import this in the prod account, the ID will contain the word dev, e.g. prod-dev-vpc not great.
Strangely enough, if you use the PrefixForAllResources property in the override parameters, then you’ll have to prepend it to all the resource IDs in the OverrideParameters as well. I.e. if in your exported bundle you have a dataset with id dataset-1 and the prefix is abcd-, then you’ll have to reference the dataset by ID abcd-dataset-1. This caught me BIG TIME while trying to import and not understanding why my override parameters are not working.
VPC Connection and DataSource override parameters are difficult to identify if not structured/tagged appropriately. Make sure you tag these resources in your AWS account or name them in such a way that you can filter the correct one out.
After you’ve managed to import a bundle, be careful, there is no ROLLBACK option! You’ll have to delete each resource one-by-one, but there is a problem with that: it seems like VPC Connections can’t be deleted permanently (Delete VPC Connection permanently)
Another thing worth mentioning is that neither the export job nor the import job deal with asset permissions. Once your bundle is imported you won’t be able to see it unless you share it from “Manage Quicksight”, or using the API to add some permissions for your admin group to each resource. (Asset Bundle Import - add permissions support)
Overall the new import/export APIs are great, they significantly helped me in trying to automate this process.
I think the ideal use-case for the APIs at the moment is to export, store zip in S3, unzip, store in source control. Use the source control to track changes and audit. Use the S3 zip for disaster recovery purposes (import to same account, same resource IDs, no prefix, no override parameters).
Importing to a different account, or to the same account with a prefix is a bit tricky at the moment. E.g. if you export dev, then upload to prod with a prefix, all resources will have the word dev in the ID. If now you export prod and try to upload to dev your resource prefix will probably be something like dev-prod-dev-43643b-.... Not great.
@Adrian_Vacaru - Thanks for the detailed feedback, this is awesome!
In terms of exporting assets can I confirm you set IncludeAllDependencies to True when exporting dashboards? If permissions weren’t included it may be by design, but I’ll have to confirm.
In regards to VPCConnections, I’d probably wouldn’t include them if the purpose of the export is to move assets from dev → test → prod. If you want to save it for Disaster Recovery then by all means include it.
In regards to PrefixForAllResources you’re right if the naming convention includes the env in the id already it gets confusing. This will work best if lowest environment, say dev, only has IDs and the prefix is used when assets are moved to higher environments.
@eperts - Yes I set the IncludeAllDependencies to True, and yes that’s a good idea to remove the VPC Connections from the bundle, I’ll try that.
Coming back to the main subject of this question (Duplicate FilterGroup IDs), it looks like the export/import APIs don’t have that issue, but I still encountered a different error:
{
"Arn": "arn:aws:quicksight:***region***:***aws_account_id***:dashboard/***dashboard_id***",
"Type": "com.amazonaws.services.quicksight.model.InvalidParameterValueException",
"Message": "FilterGroup has SINGLE_DATASET cross dataset type and contains a visual that does not match the source dataset \n\t at Filter Group d0185978-d2eb-4f36-826d-c333e3fcf1d8, \n\t at Visual 2236b11b-e42e-49e6-b16b-1ba9adcf7dca_c7e4c25b-d0dc-4579-bc1e-07e97a6fa983, \n\t at Sheet 2236b11b-e42e-49e6-b16b-1ba9adcf7dca_75698638-6b0f-48d6-bcf1-cce5ea6a21c1.",
}
My recommendation for those errors is to raise a AWS Support case. The case will be routed to the QS team and Cloud Support engineers have much more visibility than people on this forum.