Resource of type 'AWS::QuickSight::DataSource' with identifier 'MyId' did not stabilize

m0ltar · March 24, 2022, 2:27pm

Hi, I often get the error above when deploying via CloudFormation. This is extremely annoying, because the deployment fails, but then also rollback fails!

Resource handler returned message: "Resource of type 'AWS::QuickSight::DataSource' with identifier '...' did not stabilize." (RequestToken: 9ea0dbca-3fa2-951e-05db-c61d3f3c01fd, HandlerErrorCode: NotStabilized)

Then later:

Resource of type 'AWS::QuickSight::DataSource' with identifier '...' has a conflict. Reason: DataSource arn:aws:quicksight:us-east-1:123:datasource/... cannot be updated while its status is UPDATE_IN_PROGRESS; please try again (Service: QuickSight, Status Code: 409, Request ID: null, Extended Request ID: null).

lillie · May 19, 2022, 3:45pm

We resolved this issue offline and have deployed the fix

WLS-JD · October 10, 2022, 5:20pm

Hi @lillie
We are experiencing the same issue: a failed update of a Data Source during a cloudformation stack update followed by failure to Rollback due to the same error:

Resource of type ‘AWS::QuickSight::DataSource’ with identifier ‘…’ has a conflict. Reason: Permissions for DataSource arn:aws:quicksight:us-east-1:123:datasource/… cannot be updated while its status is UPDATE_IN_PROGRESS; please try again (Service: QuickSight, Status Code: 409, Request ID: null, Extended Request ID: null).

Any insight on how you fixed this issue? Is this an issue with Cloudformation?

beliash · March 30, 2023, 12:02pm

@lillie Bumping this again. Have exactly the same issue, all I am doing is adding a tag in CDK to all objects in my Stack, yet it’s suggesting I am changing permissions on a DataSource, and it hangs and eventually fails with this error.

Can somebody look into this bug please?

m0ltar · March 30, 2023, 4:02pm

Agreed, this bug is frequent, annoying and also non-descriptive.

E.g. recently I got the exact same error message when Data Source could not reach the VPC (no connection). I think it would be useful if QS would surface this error to CloudFormation, rather than produce a generic error like “Could not stabilize”.

Kristin · March 30, 2023, 10:10pm

Hi @beliash and @m0ltar Thank you for raising this, and sorry to hear that this is still an issue. I would recommend filing a case with AWS Support where we can dive into the details. Here are the steps to open a support case. If your company has someone who manages your AWS account, you might not have direct access to AWS Support and will need to raise an internal ticket to your IT team or whomever manages your AWS account. They should be able to open an AWS Support case on your behalf. Again, sorry to hear about this issue and hope this helps.

m0ltar · April 7, 2023, 6:12am

Still getting this error intermittently.

Here’s the last case of it:

Resource handler returned message: "Resource of type 'AWS::QuickSight::DataSource' with identifier 'AD-...-eu-west-1' did not stabilize." (RequestToken: f03aa506-60ea-e27e-fa6a-9746ba39e8d9, HandlerErrorCode: NotStabilized)

This time it happened when none of the infra code had changed. It was deploying a dependency upgrade to the Amplify UI, but since everything happens inside the same CloudFormation template it affects all resources.

The only thing that would change on the resources is perhaps tags, as tags get updated with the latest build ID and git commit SHA. There were no other changes to the CF template.

Kristin · April 14, 2023, 6:17am

Thank you for letting us know @m0ltar.

m0ltar · January 9, 2024, 9:37pm

Continue to hit this bug repeatedly!

CloudFormation shows status updates.

When I got into QS UI, I saw that the Data Source says has been updated 2 minutes ago.

I tested it and I can use it, it connects, and can list tables in the database.

But then a minute later CF fails with that error.

m0ltar · January 10, 2024, 10:23am

This, again, seems to be tagging-related. Changing tags alone triggers this. Can’t explain it…

m0ltar · January 10, 2024, 10:33am

Here’s my evidence:

Using AWS CDK CLI:

cdk "diff" "-e" "Foo/QST"

Stack Foo/QST (Foo-QST)
There were no differences

Then I make a small change in the CDK code and remove a couple of tags from the resource:

Tags.of(this.dataSource).remove('ad:version')
Tags.of(this.dataSource).remove('ad:build')

Do another diff, and the only difference reported are the tag changes:

cdk "diff" "-e" "Foo/QST"

Stack Foo/QST (Foo-QST)
Resources
[~] AWS::QuickSight::DataSource Foo/AD AD 
 └─ [~] Tags
     └─ @@ -1,9 +1,5 @@
        [ ] [
        [ ]   {
        [-]     "Key": "ad:build",
        [-]     "Value": "-"
        [-]   },
        [-]   {
        [ ]     "Key": "ad:module",
        [ ]     "Value": "QST"
        [ ]   },
        @@ -18,9 +14,5 @@
        [ ]   {
        [ ]     "Key": "ad:technical-contact",
        [ ]     "Value": "info@foo.io"
        [-]   },
        [-]   {
        [-]     "Key": "ad:version",
        [-]     "Value": "-"
        [ ]   }
        [ ] ]


✨  Number of stacks with differences: 1

Then I do a deploy, and it fails with:

Resource handler returned message: “Resource of type ‘AWS::QuickSight::DataSource’ with identifier ‘AD-Foo-eu-west-1’ did not stabilize.” (RequestToken: c4f98960-3d1f-752f-bde0-b2dfaa6a2298, HandlerErrorCode: NotStabilized)

m0ltar · January 10, 2024, 10:47am

I think I might have figured it out, reporting on it here in real-time

What I presume is happening is that during any update of the AWS::QuickSight::DataSource resource, the resource runs the full validation process (same as pressing the “validate” button in the UI). Even for inconsequential updates like tags.

In our case, the connection was broken, for unrelated reasons, and thus even a tagging update would fail the deployment.

I think what needs to happen is:

Do not validate on updates that do not affect the connection itself
Surface better error messages to the end user. The current error message “did not stabilize” means zero to end users. It might as well just say “error”. Given that the error message in the UI already has all of the information, please bubble it up to the end user.

The full, and useful error, as it appears in the QS UI:

Kristin · January 10, 2024, 3:57pm

Thank you for posting your findings @m0ltar!

beliash · January 29, 2024, 12:01pm

nice @m0ltar ! appreciate your acknowledgment @Kristin , but what does this mean, is the issue actively being fixed or does it need a cloudformation/Quicksight bug raising?

We currently create DataSource via cdk, and to get around this issue we need to ensure there are absolutely no elements that will inflict a change to this resource, including adding tags or setting permissions which now we have to do in our CI/CD process post-creation, it’s a real pain.

Kristin · January 29, 2024, 4:58pm

Hi @m0ltar Understood. Were you able to open up the support ticket? They are able to do a deep dive and look at logs that we do not have access to. The Customer Support Team would be best able to tell you the status once a support ticket has been opened. Unfortunately the Community Team does not have access to the Support Team tickets – since that is a different team.

m0ltar · March 14, 2024, 1:31pm

We are still experiencing this frequently, and would love to at least get a workaround solution meanwhile.

Kristin · March 18, 2024, 6:50pm

Hi @m0ltar – Thx for reaching out. Send you a message via the Community to get your account info and other details to look into this.