Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ghost updates with 5.42.0 #2647

Closed
devnev opened this issue Jul 27, 2023 · 9 comments
Closed

Ghost updates with 5.42.0 #2647

devnev opened this issue Jul 27, 2023 · 9 comments
Labels
awaiting-feedback Blocked on input from the author kind/bug Some behavior is incorrect or out of spec resolution/by-design This issue won't be fixed because the functionality is working as designed

Comments

@devnev
Copy link

devnev commented Jul 27, 2023 •
edited
Loading

What happened?

After updating to 5.42.0, we experienced ghost updates, where the resource was marked as to be updated in preview and up, but there were no changes to the properties in the diff, and the update remained even after application. A revert of only @pulumi/aws to 5.41.0 (in package.json and yarn.lock), and no other changes, fixed the issue, with the preview showing only the provider version downgrade and no other changes - and no further changes once that was applied.

Expected Behavior

No update for unmodified resources, applying update makes update go away

Steps to reproduce

The allowElbChangesPolicy in the following code was one of the resources with a ghost update

// Policy that allows ECS to assume a role
const allowEcsPolicy: aws.iam.PolicyDocument = {
  Version: '2012-10-17',
  Statement: [
    {
      Effect: 'Allow',
      Action: 'sts:AssumeRole',
      Principal: {
        Service: 'ecs.amazonaws.com',
      },
    },
  ],
};

// ECS Services declared on other repos will consume this
export const ecsServiceRole = new aws.iam.Role(
  `${stack}-role-ecs`,
  {
    description: 'Managed by Pulumi - do not edit manually',
    assumeRolePolicy: allowEcsPolicy,
  },
  {provider}
);

// Allow ECS Service role to change ELB eg target groups
// See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html
const allowElbChanges: aws.iam.PolicyDocument = {
  Version: '2012-10-17',
  Statement: [
    {
      Effect: 'Allow',
      Action: [
        'elasticloadbalancing:DeregisterInstancesFromLoadBalancer',
        'elasticloadbalancing:DeregisterTargets',
        'elasticloadbalancing:Describe*',
        'elasticloadbalancing:RegisterInstancesWithLoadBalancer',
        'elasticloadbalancing:RegisterTargets',
        'ec2:Describe*',
        'ec2:AuthorizeSecurityGroupIngress',
      ],
      Resource: '*',
    },
  ],
};

const allowElbChangesPolicy = new aws.iam.Policy(
  `${stack}-role-ecs-elb-policy`,
  {
    name: `${stack}-role-ecs-elb-policy`,
    description: 'Managey by pulumi - do not edit manually',
    policy: allowElbChanges,
  },
  {provider}
);

new aws.iam.RolePolicyAttachment(
  `${stack}-role-ecs-policy-attach`,
  {
    role: ecsServiceRole,
    policyArn: allowElbChangesPolicy.arn,
  },
  {provider}
);

Output of pulumi about

I'm not going to paste the resource URNs, but here's the rest

CLI
Version      3.72.2
Go Version   go1.20.5
Go Compiler  gc

Plugins
NAME          VERSION
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
auth0         2.23.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.41.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws           5.35.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
aws-native    0.68.0
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
awsx          1.0.2
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
docker        3.6.1
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
gcp           6.59.0
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
github        5.14.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
mongodbatlas  3.9.1
nodejs        unknown
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
postgresql    3.8.1
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0
tls           4.10.0

Host
OS       darwin
Version  12.6.5
Arch     x86_64

Found no pending operations associated with develop-infra

Backend
Name           Marks-MacBook-Pro.local
URL            s3://...
User           marknevill
Organizations

Pulumi locates its logs in /var/folders/sz/rcyd6rg51cbb4qzqsvzyn92m0000gn/T/ by default
warning: Failed to get information about the project runtime: unknown method About for service pulumirpc.LanguageRuntime
warning: Failed to get information about the Pulumi program's dependencies: unknown method GetProgramDependencies for service pulumirpc.LanguageRuntime

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@devnev devnev added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Jul 27, 2023
@guineveresaenger
Copy link
Contributor

Hi @devnev - thank you for filing this issue.

If possible, can you help us out a bit more?

When you say

and the update remained even after application.
do you mean this resource continued to be marked as an update, even after performing the ghost update?

Do you experience this behavior on the most recent version of Pulumi - could you try updating your Pulumi CLI and see if this persists?

Finally, it would really help us out if you can send us a minimum viable repro that we can run on our end to find a root cause.

@guineveresaenger guineveresaenger added awaiting-feedback Blocked on input from the author and removed needs-triage Needs attention from the triage team labels Jul 29, 2023
@danielrbradley
Copy link
Member

Hi @devnev just to add to what Guin asked - there's no provider or stack variables defined - are you using any specific configuration in the explicit provider variable?

Please could you also share what is shown in the console for you during the preview and the up - including the detailed diff.

Thanks!

@devnev
Copy link
Author

devnev commented Aug 1, 2023

Hi, thanks for the ping. We've now tried updating pulumi everywhere - notably renovate bot isn't able to update the github acction's version string automatically - and the issue has gone away.

For more background, below is the diff from the output.

~ aws:iam/policy:Policy: (update)
        [id=arn:aws:iam::123456789012:policy/develop-infra-role-ecs-elb-policy]
        [urn=urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-role-ecs-elb-policy]
        [provider=urn:pulumi:develop-infra::product-infra::pulumi:providers:aws::aws::75edc29f-b49a-4325-b2f0-0d37171a79d1]
        description: "Managey by pulumi - do not edit manually"
        name       : "develop-infra-role-ecs-elb-policy"
        path       : "/"
        policy     : (json) {
            Statement: [
                [0]: {
                    Action  : [
                        [0]: "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
                        [1]: "elasticloadbalancing:DeregisterTargets"
                        [2]: "elasticloadbalancing:Describe*"
                        [3]: "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
                        [4]: "elasticloadbalancing:RegisterTargets"
                        [5]: "ec2:Describe*"
                        [6]: "ec2:AuthorizeSecurityGroupIngress"
                    ]
                    Effect  : "Allow"
                    Resource: "*"
                }
            ]
            Version  : "2012-10-17"
        }

    ~ aws:iam/user:User: (update)
        [id=develop-infra-ci-deploy-user]
        [urn=urn:pulumi:develop-infra::product-infra::aws:iam/user:User::develop-infra-ci-deploy-user]
        [provider=urn:pulumi:develop-infra::product-infra::pulumi:providers:aws::aws::75edc29f-b49a-4325-b2f0-0d371a79d1]
        forceDestroy: false
        name        : "develop-infra-ci-deploy-user"
        path        : "/"
    ~ aws:iam/policy:Policy: (update)
        [id=arn:aws:iam::123456789012:policy/develop-infra-ci-state-update-policy-b7e9421]
        [urn=urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-ci-state-update-policy]
        [provider=urn:pulumi:develop-infra::product-infra::pulumi:providers:aws::aws::75edc29f-b49a-4325-b2f0-0d37171a79d1]
        name      : "develop-infra-ci-state-update-policy-b7e9421"
        path      : "/"
        policy    : (json) {
            Statement: [
                [0]: {
                    Action  : [
                        [0]: "s3:PutObject"
                        [1]: "s3:GetObject"
                        [2]: "s3:DeleteObject"
                    ]
                    Effect  : "Allow"
                    Resource: [
                        [0]: "arn:aws:s3:::***/.pulumi/*"
                    ]
                }
            ]
            Version  : "2012-10-17"
        }

The provider is configured as

export const provider = new aws.Provider('aws', {
  region: defaultRegion,
  defaultTags: {
    tags,
  },
});

In CI there were additional errors that we did not see when we successfully reproduced the "ghost update" issue locally:

error: 1 error occurred:
	* updating urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-role-ecs-elb-policy: 1 error occurred:
	* reading IAM Policy (arn:aws:iam::123456789012:policy/develop-infra-role-ecs-elb-policy): AccessDenied: User: arn:aws:iam::123456789012:user/develop-infra-ci-deploy-user is not authorized to perform: iam:GetPolicy on resource: policy arn:aws:iam::123456789012:policy/develop-infra-role-ecs-elb-policy because no identity-based policy allows the iam:GetPolicy action
	status code: 403, request id: 3df44795-2eea-409f-8396-4ba1b29e5ce7




error: 1 error occurred:
	* updating urn:pulumi:develop-infra::product-infra::aws:iam/user:User::develop-infra-ci-deploy-user: 1 error occurred:
	* reading IAM User (develop-infra-ci-deploy-user): AccessDenied: User: arn:aws:iam::123456789012:user/develop-infra-ci-deploy-user is not authorized to perform: iam:GetUser on resource: user develop-infra-ci-deploy-user because no identity-based policy allows the iam:GetUser action
	status code: 403, request id: 4b88c730-2990-40d5-9f2f-0a868efd409a




error: 1 error occurred:
	* updating urn:pulumi:develop-infra::product-infra::aws:iam/policy:Policy::develop-infra-ci-state-update-policy: 1 error occurred:
	* reading IAM Policy (arn:aws:iam::123456789012:policy/develop-infra-ci-state-update-policy-b7e9421): AccessDenied: User: arn:aws:iam::123456789012:user/develop-infra-ci-deploy-user is not authorized to perform: iam:GetPolicy on resource: policy arn:aws:iam::123456789012:policy/develop-infra-ci-state-update-policy-b7e9421 because no identity-based policy allows the iam:GetPolicy action
	status code: 403, request id: 6120762d-fd25-422e-ac83-35286590371a

@devnev
Copy link
Author

devnev commented Aug 1, 2023

Spoke to soon - we're seeing the issue again, even after pulumi upgrade. I'm going to downgrade @pulumi/aws again and see if that fixes it again.

@danielrbradley
Copy link
Member

danielrbradley commented Aug 2, 2023 •
edited
Loading

I've not managed to reproduce this locally myself yet. A few more follow-on questions and thoughts:

  1. Does the diff persist on subsequent updates after the initial update after the upgrade?
  2. Does the diff still occur for a repro program tha doesn't use defaultTags on the provider config?
  3. It should be possible to see the raw output from the provider's diff by using verbose logging.

@devnev
Copy link
Author

devnev commented Aug 10, 2023

We seem to have moved past this being an issue, and I can't reproduce it any more. As best as I can tell, upgrades changed the handling of default tags, that caused permission errors, and those caused updates without diffs... or the deafult tags caused updates without diffs, not quite sure on that. I'm pretty sure I also had the same diffs running locally where I have sufficient permissions, but not sure why that would have been the case. Confounding this is the various pulumi versions that were flying around.

Would you prefer we dig further, or OK with closing the issue?

@devnev
Copy link
Author

devnev commented Aug 15, 2023

Coming back to this, we're still encountering related issues, although not ghost updates. We've now downgraded back to @pulumi/aws v5.41.0. Mainly, the upgrade seemed to cause the provider's default tags to be propagated onto all sorts of resources all of a sudden, which triggered a bunch of authorization errors. Most of that could have been resolved, except for updating tags for KMS, where even a user with a policy allowing all actions on all resources gets "not authorized to perform: kms:TagResource on resource".

@mikhailshilkov
Copy link
Member

I think it all checks out - 5.42.0 introduced a fix for default tags in #2585 which it sounds like is causing downstream issues for you.

Although, I'm not exactly sure what our actions on this issue can be. Should we close it as you suggested earlier @devnev? Will you be able to work past he upgrade (and then to 6.x)?

@devnev
Copy link
Author

devnev commented Sep 14, 2023

I think we've managed to work past it, yeah. Closing, thanks

@devnev devnev closed this as completed Sep 14, 2023
@mikhailshilkov mikhailshilkov added the resolution/by-design This issue won't be fixed because the functionality is working as designed label Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting-feedback Blocked on input from the author kind/bug Some behavior is incorrect or out of spec resolution/by-design This issue won't be fixed because the functionality is working as designed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@devnev @danielrbradley @mikhailshilkov @guineveresaenger