Cloud Formation Template- to move asset from source to Target Account- Issues and resolution

dsahu · November 6, 2023, 3:33am

Hey team ,

I have been trying to create the analysis, dataset , dashboard in the target account (prod) taking assets(already present) from the dev account. Here is my cloud formation template. It is first creating a dataset, followed by template, analysis and dashboard. The datasource is already there in prod.

But it is failing while uploading even if the template is valid. saying- Invalid dataSourceArn: arn:aws:rds:us-east-1:437681213065:db:data-monitoring-20200914191548044600000002 (please refer below code)

Wanted to ask, the specifics that we are providing below is of the source account or target account?

For example: Please check code below, and could anyone let me know, what details to specify in DataSourceArn? Is it the ARN present in the source account dataset or some new arn we have to give?

Resources:
QSTCFBuildQSDataSet:
Type: AWS::QuickSight::DataSet
Properties:
DataSetId: !Join
- ‘’
- - QSTCF-DataSet
- !Ref Suffix
Name: !Join
- ‘’
- - QSTCF-DataSet
- !Ref Suffix
AwsAccountId: !Ref AWS::AccountId
PhysicalTableMap:
PhysicalTable1:
S3Source:
DataSourceArn: arn:aws:rds:us-east-1:885770418550:db:data-monitoring-20200102184109387500000001

#move to prod


BELOW IS THE TOTAL CODE: Please suggest

AWSTemplateFormatVersion: "2010-09-09"
Description: "Automated deployment of QuickSight Analysis"

Parameters:
  1stReadMe:
    Type: String
    Default: README
    Description: 'PREREQUISITES - 1) QuickSight should be setup on the aws account, 2) An user with author/admin role should be setup in QuickSight. 3)SPICE capacity should be available in the region where you are creating this stack.'

  QuickSightIdentityRegion:
    Type: String
    MinLength: 1
    Default: us-east-1
    Description: REQUIRED - QuickSight identity region (region where your users are managed; run list-users command and check the user arn if you are not sure of the identity region).

  QuickSightUser:
    Type: String
    MinLength: 1
    Default: ***
    Description: REQUIRED - User name of QuickSight author/admin from default namespace (as displayed in QuickSight admin panel). Dashboard created by this template with be shared with this user.

  AppEnvironment:
    Type: String
    Default: dev
  
  AccountVpc:
    Type: AWS::EC2::VPC::Id
    Description: Account VPC
    Default: ****

  Suffix:
    Type: String
    Description: OPTIONAL - If you need to create multiple instances of this sample on same aws account, add a short NUMERIC suffix here.

Resources:
  QSTCFBuildQSDataSet:
    Type: AWS::QuickSight::DataSet
    Properties:
      DataSetId: !Join
        - ''
        - - QSTCF-DataSet
          - !Ref Suffix
      Name: !Join
        - ''
        - - QSTCF-DataSet
          - !Ref Suffix
      AwsAccountId: !Ref AWS::AccountId
      PhysicalTableMap:
          PhysicalTable1:
            S3Source:
              DataSourceArn: arn:aws:rds:us-east-1:885770418550:db:data-monitoring-20200102184109387500000001
              InputColumns:
              - Name: datasetname
                Type: STRING
              - Name: datasettype
                Type: STRING
              - Name: bucketname
                Type: STRING
              - Name: datasetcreatetime
                Type: STRING
              - Name: arrival_time
                Type: STRING
              - Name: time
                Type: STRING
              - Name: fileparsedate
                Type: STRING
              - Name: date
                Type: STRING
              - Name: carr_acctg_cd
                Type: STRING
              - Name: CARR_ACCTG_CD
                Type: STRING
              - Name: CARR_NM_1
                Type: STRING
              - Name: MIN_HOUR
                Type: STRING
              - Name: MinHour in Minutes from Zero hour
                Type: STRING
              - Name: MAX_HOUR
                Type: STRING
              - Name: MaxHour in Minutes from Zero hour
                Type: STRING
              # UploadSettings:
              #   ContainsHeader: true
              #   Delimiter: ','
              #   Format: CSV
              #   StartFromRow: 1
              #   TextQualifier: DOUBLE_QUOTE
      Permissions:
          - Principal: !Join
              - ''
              - - 'arn:aws:quicksight:'
                - !Ref QuickSightIdentityRegion
                - ':'
                - !Ref AWS::AccountId
                - ':user/default/'
                - !Ref QuickSightUser
            Actions:
              - quicksight:UpdateDataSetPermissions
              - quicksight:DescribeDataSet
              - quicksight:DescribeDataSetPermissions
              - quicksight:PassDataSet
              - quicksight:DescribeIngestion
              - quicksight:ListIngestions
              - quicksight:UpdateDataSet
              - quicksight:DeleteDataSet
              - quicksight:CreateIngestion
              - quicksight:CancelIngestion
      ImportMode: SPICE

  QSTCFBuildQSTemplate:
    Type: 'AWS::QuickSight::Template'
    Properties:
      TemplateId: "missing_files_parse_date_compare_date_trend-Template"
      Name: "missing_files_parse_date_compare_date_trend-Template"
      AwsAccountId: !Ref AWS::AccountId
      SourceEntity:
        SourceTemplate:
          Arn: 'arn:aws:quicksight:us-east-1:885770418550:template/missing_files_parse_date_compare_date_trend-Template'          
      Permissions:
        - Principal: !Join 
            - ''
            - - 'arn:aws:quicksight:'
              - !Ref QuickSightIdentityRegion
              - ':'
              - !Ref 'AWS::AccountId'
              - ':user/default/'
              - !Ref QuickSightUser
          Actions:
            - 'quicksight:DescribeTemplate'
      VersionDescription: Initial version - Copied over from AWS account
      
  QSRSBuildQSAnalysis:
    Type: 'AWS::QuickSight::Analysis'
    Properties:
      AnalysisId: 'fdb55fdb-831c-472e-805d-e00b9051d10a'
      Name: 'missing_files_parse_date_compare_date_trend analysis'
      AwsAccountId: !Ref AWS::AccountId
      SourceEntity:
        SourceTemplate:
          Arn: !GetAtt  QSTCFBuildQSTemplate.Arn
          DataSetReferences:
            - DataSetPlaceholder: 'missing_files_parse_date_compare_date_trend'
              DataSetArn: !GetAtt  QSTCFBuildQSTemplate.Arn
      Permissions:
        - Principal: !Join 
            - ''
            - - 'arn:aws:quicksight:'
              - !Ref QuickSightIdentityRegion
              - ':'
              - !Ref 'AWS::AccountId'
              - ':user/default/'
              - !Ref QuickSightUser
          Actions:
            - 'quicksight:RestoreAnalysis'
            - 'quicksight:UpdateAnalysisPermissions'
            - 'quicksight:DeleteAnalysis'
            - 'quicksight:DescribeAnalysisPermissions'
            - 'quicksight:QueryAnalysis'
            - 'quicksight:DescribeAnalysis'
            - 'quicksight:UpdateAnalysis'
            
  QSTCFBuildQSDashboard:
    Type: AWS::QuickSight::Dashboard
    Properties:
      DashboardId: !Join
        - ''
        - - QSTCF-Dashboard
          - !Ref Suffix
      Name: !Join
        - ''
        - - QSTCF-Dashboard
          - !Ref Suffix
      AwsAccountId: !Ref AWS::AccountId
      SourceEntity:
        SourceTemplate:
          Arn: !GetAtt QSTCFBuildQSTemplate.Arn
          DataSetReferences:
            - DataSetPlaceholder: missing_files_parse_date_compare_date_trend
              DataSetArn: !GetAtt QSTCFBuildQSDataSet.Arn
      Permissions:
        - Principal: !Join
            - ''
            - - 'arn:aws:quicksight:'
              - !Ref QuickSightIdentityRegion
              - ':'
              - !Ref AWS::AccountId
              - ':user/default/'
              - !Ref QuickSightUser
          Actions:
            - quicksight:DescribeDashboard
            - quicksight:ListDashboardVersions
            - quicksight:UpdateDashboardPermissions
            - quicksight:QueryDashboard
            - quicksight:UpdateDashboard
            - quicksight:DeleteDashboard
            - quicksight:DescribeDashboardPermissions
            - quicksight:UpdateDashboardPublishedVersion
      # ThemeArn: !GetAtt QSTCFBuildQSTheme.Arn
      # DashboardPublishOptions:
      #   AdHocFilteringOption:
      #     AvailabilityStatus: DISABLED

Waiting for a response.

Really want to see, if after uploading this template i will be able to see a dataset/analysis and a dashboard being populated in the target account. Or we have to specify something more for the quicksight.

Thank you.

andres007 · November 6, 2023, 8:49am

Hi,

I think you might be using the wrong type of source for your PhysicalTableMap, since you are using S3Source but pointing to an RDS ARN. You should use

Hope this helps!

dsahu · November 7, 2023, 5:13pm

Hi @andres007 and team,

I modified it (changing s3 source to RelationalTable as I am connecting via RDS) and streamlined it to just to create a dataset in the target account:

AWSTemplateFormatVersion: "2010-09-09"
Description: "Automated deployment of QuickSight Analysis"

Parameters:
  1stReadMe:
    Type: String
    Default: README
    Description: 'PREREQUISITES - 1) QuickSight should be setup on the aws account, 2) An user with author/admin role should be setup in QuickSight. 3)SPICE capacity should be available in the region where you are creating this stack.'

  QuickSightIdentityRegion:
    Type: String
    MinLength: 1
    Default: us-east-1
    Description: REQUIRED - QuickSight identity region (region where your users are managed; run list-users command and check the user arn if you are not sure of the identity region).

  QuickSightUser:
    Type: String
    MinLength: 1
    Default: ***
    Description: REQUIRED - User name of QuickSight author/admin from default namespace (as displayed in QuickSight admin panel). Dashboard created by this template with be shared with this user.

  AppEnvironment:
    Type: String
    Default: dev
  
  AccountVpc:
    Type: AWS::EC2::VPC::Id
    Description: Account VPC
    Default: vpc-2c193e57

  Suffix:
    Type: String
    Description: OPTIONAL - If you need to create multiple instances of this sample on same aws account, add a short NUMERIC suffix here.

Resources:
  QSTCFBuildQSDataSet:
    Type: AWS::QuickSight::DataSet
    Properties:
      DataSetId: !Join
        - ''
        - - QSTCF-DataSet
          - !Ref Suffix
      Name: !Join
        - ''
        - - QSTCF-DataSet
          - !Ref Suffix
      AwsAccountId: !Ref AWS::AccountId
      PhysicalTableMap:
          PhysicalTable1:
            RelationalTable:
              **DataSourceArn: arn:aws:rds:us-east-1:885770418550:db:data-monitoring-20200102184109387500000001**
              InputColumns:
              - Name: datasetid
                Type: DECIMAL
              - Name: datasetname
                Type: STRING
              - Name: datasettype
                Type: STRING
              Name: Dataset
      Permissions:
          - Principal: !Join
              - ''
              - - 'arn:aws:quicksight:'
                - !Ref QuickSightIdentityRegion
                - ':'
                - !Ref AWS::AccountId
                - ':user/default/'
                - !Ref QuickSightUser
            Actions:
              - quicksight:UpdateDataSetPermissions
              - quicksight:DescribeDataSet
              - quicksight:DescribeDataSetPermissions
              - quicksight:PassDataSet
              - quicksight:DescribeIngestion
              - quicksight:ListIngestions
              - quicksight:UpdateDataSet
              - quicksight:DeleteDataSet
              - quicksight:CreateIngestion
              - quicksight:CancelIngestion
      ImportMode: SPICE

Error:

Resource handler returned message: “Invalid request provided: Invalid dataSourceArn: arn:aws:rds:us-east-1:885770418550:db:data-monitoring-20200102184109387500000001 (Service: QuickSight, Status Code: 400, Request ID: 75bc1310-c0fd-4ca6-bd4d-fe89251230af)” (RequestToken: 26c85291-901c-37fd-68f4-7f5e4fb14a05, HandlerErrorCode: InvalidRequest)

Could anyone please let me know, why even after providing the exact ARN od the target account, I am still getting the invalid arn issue.

Is there anything That I am missing.

My objective- To see the dataset “DATASET” in the target account being populated.

Thank you for your help.

andres007 · November 8, 2023, 1:01pm

Hi,

The data source arn is a quicksight data source arn, not the ARN of the RDS DB. Should be something like this.

arn:aws:quicksight:us-west-2:###########:datasource/3f979181-####-####-####-90be58497350

So you should also be creating the DataSource in the CF, and use that ARN as the data source for your data set if you dont have it created already on the prod account, if so, you can use the arn of that datasource for your CFN template.

Hope this helps!

dsahu · November 8, 2023, 5:27pm

Thank you so much @andres007 for clearing this, I will follow the same, This cleared my doubt on the arn value that I was giving incorrectly.

Also, one more issue I am facing here,

While I am creating an analysis (in the PROD account-target account), from my dev account (source)-
The code is below:

Question- I am using the source template ARN (ARN of the dev account quicksight template) and the dataset ARN is of the target account , as you can see below - (dataset arn) as I created the DATASET directly in the target(prod) account first (dataset I didn’t create via cft, as we wanted real time data from the RDS)

there are 2 different account ids.

So, while uploading to CF, to see if analysis is created or not - the error iam getting is below-

i.e I am getting access denied permission, so is it that, we have to set some permission for the cross account connectivity? Can you please confirm, or the code itself needs correction?

Should I use- this- aws quicksight update-template-permissions --aws-account-id TARGET ACCOUNT ID --template-id “TARGET TEMPLATE ID” --grant-permissions file://TemplatePermission.json ??

#move to prod


AWSTemplateFormatVersion: "2010-09-09"
Description: "Automated deployment of QuickSight Analysis"

Parameters:
  1stReadMe:
    Type: String
    Default: README
    Description: 'PREREQUISITES - 1) QuickSight should be setup on the aws account, 2) An user with author/admin role should be setup in QuickSight. 3)SPICE capacity should be available in the region where you are creating this stack.'

  QuickSightIdentityRegion:
    Type: String
    MinLength: 1
    Default: us-east-1
    Description: REQUIRED - QuickSight identity region (region where your users are managed; run list-users command and check the user arn if you are not sure of the identity region).

  QuickSightUser:
    Type: String
    MinLength: 1
    Default: ****
    Description: REQUIRED - User name of QuickSight author/admin from default namespace (as displayed in QuickSight admin panel). Dashboard created by this template with be shared with this user.

  AppEnvironment:
    Type: String
    Default: dev
  
  AccountVpc:
    Type: AWS::EC2::VPC::Id
    Description: Account VPC
    Default: *****

  Suffix:
    Type: String
    Description: OPTIONAL - If you need to create multiple instances of this sample on same aws account, add a short NUMERIC suffix here.

Resources:    
  QSRSBuildQSAnalysis:
    Type: 'AWS::QuickSight::Analysis'
    Properties:
      AnalysisId: 'missing_file_analysis_cft'
      Name: 'missing_files_parse_date_compare_date_trend_analysis_cft'
      AwsAccountId: !Ref AWS::AccountId
      SourceEntity:
        SourceTemplate:
          Arn: "arn:aws:quicksight:us-east-1:885770418550:template/missing_files_parse_date_compare_date_trend_Template"
          DataSetReferences:
            - DataSetPlaceholder: 'dataset_missing_files'
              DataSetArn: "arn:aws:quicksight:us-east-1:437681213065:dataset/41e3ebd5-2cc6-4d71-af5c-d6074649f9fe"
      Permissions:
        - Principal: !Join 
            - ''
            - - 'arn:aws:quicksight:'
              - !Ref QuickSightIdentityRegion
              - ':'
              - !Ref 'AWS::AccountId'
              - ':user/default/'
              - !Ref QuickSightUser
          Actions:
            - 'quicksight:RestoreAnalysis'
            - 'quicksight:UpdateAnalysisPermissions'
            - 'quicksight:DeleteAnalysis'
            - 'quicksight:DescribeAnalysisPermissions'
            - 'quicksight:QueryAnalysis'
            - 'quicksight:DescribeAnalysis'
            - 'quicksight:UpdateAnalysis'

Appreciate your response in this as I am stuck and needed help in this.

Thank you again.

andres007 · November 9, 2023, 9:12am

Hello,

To grant cross account permisions to the template that seems to be the right way, I found this re:Post that can help you setting that up.

Having said that, depending on how strict the dev/prod separation in your company is, you might want to look at copying the template over to the prod account as part of the deployment process to make sure the template cannot be changed on the dev account which could impact production.

Hope this helps!

dsahu · November 16, 2023, 2:17pm

Thank you so much @andres007 I am successful now in deploying in PROD using cloudformation template, The key point is we have to make sure the template id, for which permissions granted matches to the template id in the creation of analysis CFT . This was the place where I was making the mistake. Now it is resolved.

However I have one more issue, would really be helpful if you can provide some insights.

As discussed,
I have been successful in doing the same, and then incorporating terraform to run the pipeline that upon execution, uploads the Cloudformation template , creates the stack and deploys my assets in the pROD. It is Successful!!

However, I am not able to Provide Access Permission to Multiple users at the same time through CloudFormation.

We have a template in json where we have to provide permission from our DEV to the destination account below as you can see

[
    {
      "Principal": "arn:aws:iam::target_Account_ID:root",
      "Actions": [
        "quicksight:UpdateTemplatePermissions",
        "quicksight:DescribeTemplate"
      ]
    }
  ]

However, the quicksight user name which has been created via terraform , is unable to see the assets, even though the cloudformation stack is created and successful.

Is there a way, where we can work on the permission part, if multiple users can be given permissions at the same time then the resources might show up for everyone in prod after deployment. Please let me know.

Thank you so much.

Appreciate your help in this.