Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic replacing ec2 instance after user data is updated #2904

Closed
VenelinMartinov opened this issue Oct 19, 2023 · 8 comments
Closed

Panic replacing ec2 instance after user data is updated #2904

VenelinMartinov opened this issue Oct 19, 2023 · 8 comments
Assignees
Labels
bug/diff kind/bug related to Pulumi generating wrong diffs on preview or up. impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed

Comments

@VenelinMartinov
Copy link
Contributor

VenelinMartinov commented Oct 19, 2023

What happened?

I ran pulumi up on a project and got a panic during ec2 instance replacement.
I originally started with the vm-aws-python template and was updating the user data of the ec2 instance when I got the panic.

Example

I can reliably reproduce the panic in the original project but I wasn't able to reproduce it in a new project.

Error in the output of pulumi up:

     Type                 Name              Status                  Info
     Type                 Name              Status                  Info
     pulumi:pulumi:Stack  video_search-dev  **failed**              1 error; 36 messages
 ~   └─ aws:ec2:Instance  server            **updating failed**     [diff: ~userData]; 1 error

Diagnostics:
  aws:ec2:Instance (server):
    error: error reading from server: EOF

  pulumi:pulumi:Stack (video_search-dev):
    panic: interface conversion: interface {} is nil, not map[string]interface {}
    goroutine 144 [running]:
    github.com/hashicorp/go-cty/cty.Value.GetAttr({{{0x11120dd70?, 0x140050da338?}}, {0x0?, 0x0?}}, {0x10cca7e0f, 0x8})
    	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/cty/value_ops.go:711 +0x258
    github.com/hashicorp/terraform-provider-aws/internal/provider.tagsResourceInterceptor.run({0x14000c96960?, 0x1110b1060?, 0x1110b1058?}, {0x11120d948, 0x14002af98f0}, {0x11121e558, 0x14000a5be80}, {0x11104fa00?, 0x14002207a40?}, 0x1, ...)
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:250 +0xea4
    github.com/hashicorp/terraform-provider-aws/internal/provider.New.(*wrappedResource).Update.interceptedHandler[...].func10(0x0?, {0x11104fa00?, 0x14002207a40?})
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:100 +0x11c
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0x11120da28?, {0x11120da28?, 0x11bc965e0?}, 0xd?, {0x11104fa00?, 0x14002207a40?})
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:812 +0x64
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0x140010a20e0, {0x11120da28, 0x11bc965e0}, 0x140069008f0, 0x14000819400, {0x11104fa00, 0x14002207a40})
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:924 +0x658
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0x14000127c80?, {0x14000c930b0?, 0x14004f0e7d0?, 0x0?}}, {0x10ccc0613, 0xc}, {0x11120de58?, 0x14001c914d0}, {0x11121e020?, 0x14000819400})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:100 +0x140
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0x140007d2580, {0x11120d948?, 0x140074d5b00?}, 0x14008496870)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:975 +0x6d0
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Update.func1({0x11123c378?, 0x140007d2580?})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:356 +0x3c
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.resourceMethod[...](0x14008504c80?, 0x14004f9d788, 0x14004f9d768?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:303 +0xc4
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Update(0x14004f9d7c8?, {0x11120d948?, 0x140074d5b00?}, 0x10ebb9100?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:355 +0x5c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0x11120d948, 0x140074d5b00}, {0x110c5f460?, 0x14008496870})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:609 +0x74
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x11120d948, 0x140074d5200}, {0x110c5f460, 0x14008496870}, 0x14003f50ae0, 0x14001c90af8)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x2e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0x110ebc8a0?, 0x14008504c80}, {0x11120d948, 0x140074d5200}, 0x14003f0d810, 0x140083ec160)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:611 +0x12c
    google.golang.org/grpc.(*Server).processUnaryRPC(0x14000130000, {0x1112203c0, 0x14004d4c820}, 0x14004c7e6c0, 0x140029b7bf0, 0x11bc42ff0, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1360 +0xc04
    google.golang.org/grpc.(*Server).handleStream(0x14000130000, {0x1112203c0, 0x14004d4c820}, 0x14004c7e6c0, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1737 +0x7c4
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:982 +0x84
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 86
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:980 +0x154

    error: update failed

Outputs:
  - hostname: "ec2-34-220-171-98.us-west-2.compute.amazonaws.com"
  - ip      : "34.220.171.98"
  - url     : "http://ec2-34-220-171-98.us-west-2.compute.amazonaws.com"

Resources:
    8 unchanged

Duration: 13s

I ran pulumi stack export to get the current state of the stack:

``` { "version": 3, "deployment": { "manifest": { "time": "2023-10-18T19:34:44.481019+03:00", "magic": "65806507328cc506774f090af301e8d1d61197a34ee66d0cc264fc92f5dde976", "version": "v3.89.0" }, "secrets_providers": { "type": "service", "state": { "url": "https://api.pulumi.com", "owner": "venelin-pulumi-corp", "project": "video_search", "stack": "dev" } }, "resources": [ { "urn": "urn:pulumi:dev::video_search::pulumi:providers:aws::default", "custom": true, "id": "c9766f20-a3ce-442c-892b-b39d5269f79e", "type": "pulumi:providers:aws", "inputs": { "region": "us-west-2", "skipCredentialsValidation": "false", "skipMetadataApiCheck": "true", "skipRegionValidation": "true", "version": "6.5.0" }, "outputs": { "region": "us-west-2", "skipCredentialsValidation": "false", "skipMetadataApiCheck": "true", "skipRegionValidation": "true", "version": "6.5.0" }, "created": "2023-10-17T09:31:00.795448Z", "modified": "2023-10-17T09:31:00.795448Z" }, { "urn": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "custom": false, "type": "pulumi:pulumi:Stack", "outputs": { "hostname": "ec2-34-220-171-98.us-west-2.compute.amazonaws.com", "ip": "34.220.171.98", "url": "http://ec2-34-220-171-98.us-west-2.compute.amazonaws.com" }, "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi/runtime/stack.py#137" }, { "urn": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0", "custom": true, "id": "1134e720-e72f-496e-a8f6-691b2002752f", "type": "pulumi:providers:aws", "inputs": { "region": "us-west-2", "skipCredentialsValidation": "false", "skipMetadataApiCheck": "true", "skipRegionValidation": "true", "version": "6.5.0" }, "outputs": { "region": "us-west-2", "skipCredentialsValidation": "false", "skipMetadataApiCheck": "true", "skipRegionValidation": "true", "version": "6.5.0" }, "created": "2023-10-17T09:31:06.243062Z", "modified": "2023-10-17T09:31:06.243062Z" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/keyPair:KeyPair::keypair", "custom": true, "id": "keypair-a98ff69", "type": "aws:ec2/keyPair:KeyPair", "inputs": { "__defaults": [ "keyName" ], "keyName": "keypair-a98ff69", "publicKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIELgS7M+lBg/8WOnlWkrOOWjc7juuq5b8L+wgWct/Yia" }, "outputs": { "__meta": "{\"schema_version\":\"1\"}", "arn": "arn:aws:ec2:us-west-2:616138583583:key-pair/keypair-a98ff69", "fingerprint": "LmpqV4AMdzTkeDJ42Ep2tDA5Di98VPIzArXa10ZzVlQ=", "id": "keypair-a98ff69", "keyName": "keypair-a98ff69", "keyNamePrefix": "", "keyPairId": "key-0a85a888de3ab757b", "keyType": "ed25519", "publicKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIELgS7M+lBg/8WOnlWkrOOWjc7juuq5b8L+wgWct/Yia", "tags": {}, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABAHiIA1cWFMsgjBgtXYWrsFXMs5X1J4bz4rCljMK3RA==" } }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "publicKey": null }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T10:07:53.080654Z", "modified": "2023-10-17T10:07:53.080654Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/key_pair.py#334" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc", "custom": true, "id": "vpc-08a1eb27f9badc04a", "type": "aws:ec2/vpc:Vpc", "inputs": { "__defaults": [], "cidrBlock": "10.0.0.0/16", "enableDnsHostnames": true, "enableDnsSupport": true }, "outputs": { "__meta": "{\"schema_version\":\"1\"}", "arn": "arn:aws:ec2:us-west-2:616138583583:vpc/vpc-08a1eb27f9badc04a", "assignGeneratedIpv6CidrBlock": false, "cidrBlock": "10.0.0.0/16", "defaultNetworkAclId": "acl-0deaa4111b2ad57ce", "defaultRouteTableId": "rtb-02cde28d939e9b4f3", "defaultSecurityGroupId": "sg-0f45d4b10dea37504", "dhcpOptionsId": "dopt-1649d26e", "enableDnsHostnames": true, "enableDnsSupport": true, "enableNetworkAddressUsageMetrics": false, "id": "vpc-08a1eb27f9badc04a", "instanceTenancy": "default", "ipv6AssociationId": "", "ipv6CidrBlock": "", "ipv6CidrBlockNetworkBorderGroup": "", "ipv6IpamPoolId": "", "ipv6NetmaskLength": 0, "mainRouteTableId": "rtb-02cde28d939e9b4f3", "ownerId": "616138583583", "tags": {}, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABAHrDPZ0C7PvkicLutjyvL+7v/p9Wqat+khsYx9cldA==" } }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "cidrBlock": null, "enableDnsHostnames": null, "enableDnsSupport": null }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T09:31:22.181099Z", "modified": "2023-10-17T09:31:22.181099Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/vpc.py#765" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/internetGateway:InternetGateway::gateway", "custom": true, "id": "igw-0935059d259c619f5", "type": "aws:ec2/internetGateway:InternetGateway", "inputs": { "__defaults": [], "vpcId": "vpc-08a1eb27f9badc04a" }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":1200000000000,\"delete\":1200000000000,\"update\":1200000000000}}", "arn": "arn:aws:ec2:us-west-2:616138583583:internet-gateway/igw-0935059d259c619f5", "id": "igw-0935059d259c619f5", "ownerId": "616138583583", "tags": {}, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABACzzM5wt5BQbXtrc0/4J30OTgeHh1e3bwAhdS5QBmg==" }, "vpcId": "vpc-08a1eb27f9badc04a" }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "dependencies": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ], "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "vpcId": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ] }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T09:31:24.978204Z", "modified": "2023-10-17T09:31:24.978204Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/internet_gateway.py#280" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/subnet:Subnet::subnet", "custom": true, "id": "subnet-074006f6b09c71553", "type": "aws:ec2/subnet:Subnet", "inputs": { "__defaults": [ "assignIpv6AddressOnCreation", "enableDns64", "enableResourceNameDnsARecordOnLaunch", "enableResourceNameDnsAaaaRecordOnLaunch", "ipv6Native" ], "assignIpv6AddressOnCreation": false, "cidrBlock": "10.0.1.0/24", "enableDns64": false, "enableResourceNameDnsARecordOnLaunch": false, "enableResourceNameDnsAaaaRecordOnLaunch": false, "ipv6Native": false, "mapPublicIpOnLaunch": true, "vpcId": "vpc-08a1eb27f9badc04a" }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"delete\":1200000000000},\"schema_version\":\"1\"}", "arn": "arn:aws:ec2:us-west-2:616138583583:subnet/subnet-074006f6b09c71553", "assignIpv6AddressOnCreation": false, "availabilityZone": "us-west-2b", "availabilityZoneId": "usw2-az1", "cidrBlock": "10.0.1.0/24", "customerOwnedIpv4Pool": "", "enableDns64": false, "enableLniAtDeviceIndex": 0, "enableResourceNameDnsARecordOnLaunch": false, "enableResourceNameDnsAaaaRecordOnLaunch": false, "id": "subnet-074006f6b09c71553", "ipv6CidrBlock": "", "ipv6CidrBlockAssociationId": "", "ipv6Native": false, "mapCustomerOwnedIpOnLaunch": false, "mapPublicIpOnLaunch": true, "outpostArn": "", "ownerId": "616138583583", "privateDnsHostnameTypeOnLaunch": "ip-name", "tags": {}, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABAMtNYGrxHIV5pjz1fT1gwgp96pz8X9ogYew0EhjPxg==" }, "vpcId": "vpc-08a1eb27f9badc04a" }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "dependencies": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ], "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "cidrBlock": null, "mapPublicIpOnLaunch": null, "vpcId": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ] }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T09:31:35.365647Z", "modified": "2023-10-17T09:31:35.365647Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/subnet.py#815" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/securityGroup:SecurityGroup::secGroup", "custom": true, "id": "sg-0977e265450e07c62", "type": "aws:ec2/securityGroup:SecurityGroup", "inputs": { "__defaults": [ "name", "revokeRulesOnDelete" ], "description": "Enable HTTP access", "egress": [ { "__defaults": [ "self" ], "cidrBlocks": [ "0.0.0.0/0" ], "fromPort": 0, "protocol": "-1", "self": false, "toPort": 0 } ], "ingress": [ { "__defaults": [ "self" ], "cidrBlocks": [ "0.0.0.0/0" ], "fromPort": 80, "protocol": "tcp", "self": false, "toPort": 80 }, { "__defaults": [ "self" ], "cidrBlocks": [ "0.0.0.0/0" ], "fromPort": 22, "protocol": "tcp", "self": false, "toPort": 22 } ], "name": "secGroup-86b3691", "revokeRulesOnDelete": false, "vpcId": "vpc-08a1eb27f9badc04a" }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"delete\":900000000000},\"schema_version\":\"1\"}", "arn": "arn:aws:ec2:us-west-2:616138583583:security-group/sg-0977e265450e07c62", "description": "Enable HTTP access", "egress": [ { "cidrBlocks": [ "0.0.0.0/0" ], "description": "", "fromPort": 0, "ipv6CidrBlocks": [], "prefixListIds": [], "protocol": "-1", "securityGroups": [], "self": false, "toPort": 0 } ], "id": "sg-0977e265450e07c62", "ingress": [ { "cidrBlocks": [ "0.0.0.0/0" ], "description": "", "fromPort": 80, "ipv6CidrBlocks": [], "prefixListIds": [], "protocol": "tcp", "securityGroups": [], "self": false, "toPort": 80 }, { "cidrBlocks": [ "0.0.0.0/0" ], "description": "", "fromPort": 22, "ipv6CidrBlocks": [], "prefixListIds": [], "protocol": "tcp", "securityGroups": [], "self": false, "toPort": 22 } ], "name": "secGroup-86b3691", "namePrefix": "", "ownerId": "616138583583", "revokeRulesOnDelete": false, "tags": {}, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABAAL+SXpGphFND2lIOjo+rEe57i8FwUUU8j2ohizFvQ==" }, "vpcId": "vpc-08a1eb27f9badc04a" }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "dependencies": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ], "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "description": null, "egress": null, "ingress": null, "vpcId": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ] }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T09:31:27.651334Z", "modified": "2023-10-17T10:07:53.88837Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/security_group.py#689" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/routeTable:RouteTable::routeTable", "custom": true, "id": "rtb-05042aba1bdca7fe1", "type": "aws:ec2/routeTable:RouteTable", "inputs": { "__defaults": [], "routes": [ { "__defaults": [], "cidrBlock": "0.0.0.0/0", "gatewayId": "igw-0935059d259c619f5" } ], "vpcId": "vpc-08a1eb27f9badc04a" }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":300000000000,\"delete\":300000000000,\"update\":120000000000}}", "arn": "arn:aws:ec2:us-west-2:616138583583:route-table/rtb-05042aba1bdca7fe1", "id": "rtb-05042aba1bdca7fe1", "ownerId": "616138583583", "propagatingVgws": [], "routes": [ { "carrierGatewayId": "", "cidrBlock": "0.0.0.0/0", "coreNetworkArn": "", "destinationPrefixListId": "", "egressOnlyGatewayId": "", "gatewayId": "igw-0935059d259c619f5", "ipv6CidrBlock": "", "localGatewayId": "", "natGatewayId": "", "networkInterfaceId": "", "transitGatewayId": "", "vpcEndpointId": "", "vpcPeeringConnectionId": "" } ], "tags": {}, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABAC6LlVxgn5SLroysooaLx7an+jmz5iAA8A8EXlSJuw==" }, "vpcId": "vpc-08a1eb27f9badc04a" }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "dependencies": [ "urn:pulumi:dev::video_search::aws:ec2/internetGateway:InternetGateway::gateway", "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ], "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "routes": [ "urn:pulumi:dev::video_search::aws:ec2/internetGateway:InternetGateway::gateway" ], "vpcId": [ "urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc" ] }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T09:31:27.684451Z", "modified": "2023-10-17T09:31:27.684451Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/route_table.py#465" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/routeTableAssociation:RouteTableAssociation::routeTableAssociation", "custom": true, "id": "rtbassoc-01f7e2c6cc2b7f824", "type": "aws:ec2/routeTableAssociation:RouteTableAssociation", "inputs": { "__defaults": [], "routeTableId": "rtb-05042aba1bdca7fe1", "subnetId": "subnet-074006f6b09c71553" }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":300000000000,\"delete\":300000000000,\"update\":120000000000}}", "gatewayId": "", "id": "rtbassoc-01f7e2c6cc2b7f824", "routeTableId": "rtb-05042aba1bdca7fe1", "subnetId": "subnet-074006f6b09c71553" }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "dependencies": [ "urn:pulumi:dev::video_search::aws:ec2/routeTable:RouteTable::routeTable", "urn:pulumi:dev::video_search::aws:ec2/subnet:Subnet::subnet" ], "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "routeTableId": [ "urn:pulumi:dev::video_search::aws:ec2/routeTable:RouteTable::routeTable" ], "subnetId": [ "urn:pulumi:dev::video_search::aws:ec2/subnet:Subnet::subnet" ] }, "created": "2023-10-17T09:31:37.004712Z", "modified": "2023-10-17T09:31:37.004712Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/route_table_association.py#246" }, { "urn": "urn:pulumi:dev::video_search::aws:ec2/instance:Instance::server", "custom": true, "id": "i-0f38265656a7c6ba9", "type": "aws:ec2/instance:Instance", "inputs": { "__defaults": [ "getPasswordData", "sourceDestCheck", "userDataReplaceOnChange" ], "ami": "ami-0025f0db847eb6254", "getPasswordData": false, "instanceType": "g4dn.xlarge", "keyName": "keypair-a98ff69", "sourceDestCheck": true, "subnetId": "subnet-074006f6b09c71553", "tags": { "Name": "webserver", "__defaults": [] }, "userData": "#!/bin/bash\necho \"Hello, World from Pulumi!\" \u003e index.html\nnohup python -m SimpleHTTPServer 80 \u0026\n", "userDataReplaceOnChange": false, "vpcSecurityGroupIds": [ "sg-0977e265450e07c62" ] }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"delete\":1200000000000,\"update\":600000000000},\"schema_version\":\"1\"}", "ami": "ami-0025f0db847eb6254", "arn": "arn:aws:ec2:us-west-2:616138583583:instance/i-0f38265656a7c6ba9", "associatePublicIpAddress": true, "availabilityZone": "us-west-2b", "capacityReservationSpecification": { "capacityReservationPreference": "open", "capacityReservationTarget": null }, "cpuCoreCount": 2, "cpuOptions": { "amdSevSnp": "", "coreCount": 2, "threadsPerCore": 2 }, "cpuThreadsPerCore": 2, "creditSpecification": null, "disableApiStop": false, "disableApiTermination": false, "ebsBlockDevices": [], "ebsOptimized": false, "enclaveOptions": { "enabled": false }, "ephemeralBlockDevices": [], "getPasswordData": false, "hibernation": false, "hostId": "", "iamInstanceProfile": "", "id": "i-0f38265656a7c6ba9", "instanceInitiatedShutdownBehavior": "stop", "instanceLifecycle": "", "instanceMarketOptions": null, "instanceState": "running", "instanceType": "g4dn.xlarge", "ipv6AddressCount": 0, "ipv6Addresses": [], "keyName": "keypair-a98ff69", "launchTemplate": null, "maintenanceOptions": { "autoRecovery": "default" }, "metadataOptions": { "httpEndpoint": "enabled", "httpProtocolIpv6": "disabled", "httpPutResponseHopLimit": 1, "httpTokens": "optional", "instanceMetadataTags": "disabled" }, "monitoring": false, "networkInterfaces": [], "outpostArn": "", "passwordData": "", "placementGroup": "", "placementPartitionNumber": 0, "primaryNetworkInterfaceId": "eni-05796f45d86e526cb", "privateDns": "ip-10-0-1-182.us-west-2.compute.internal", "privateDnsNameOptions": { "enableResourceNameDnsARecord": false, "enableResourceNameDnsAaaaRecord": false, "hostnameType": "ip-name" }, "privateIp": "10.0.1.182", "publicDns": "ec2-34-220-171-98.us-west-2.compute.amazonaws.com", "publicIp": "34.220.171.98", "rootBlockDevice": { "deleteOnTermination": true, "deviceName": "/dev/xvda", "encrypted": false, "iops": 100, "kmsKeyId": "", "tags": {}, "throughput": 0, "volumeId": "vol-0a06f691ec9e1b596", "volumeSize": 8, "volumeType": "gp2" }, "secondaryPrivateIps": [], "securityGroups": [], "sourceDestCheck": true, "spotInstanceRequestId": "", "subnetId": "subnet-074006f6b09c71553", "tags": { "Name": "webserver" }, "tagsAll": { "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270", "ciphertext": "AAABAJoDV41D+9s275qTUpWV22P5j04i3/cZZDW2p5KqFqM8CDxQasUpq96DsT06tDC1rw==" }, "tenancy": "default", "userData": "8ad164e8f8824306d2c5971b0b7bd234b62e284a", "userDataReplaceOnChange": false, "vpcSecurityGroupIds": [ "sg-0977e265450e07c62" ] }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "dependencies": [ "urn:pulumi:dev::video_search::aws:ec2/keyPair:KeyPair::keypair", "urn:pulumi:dev::video_search::aws:ec2/securityGroup:SecurityGroup::secGroup", "urn:pulumi:dev::video_search::aws:ec2/subnet:Subnet::subnet" ], "initErrors": [ "updating urn:pulumi:dev::video_search::aws:ec2/instance:Instance::server: 1 error occurred:\n\t* updating EC2 Instance (i-0f38265656a7c6ba9) user data: stopping EC2 Instance: InvalidInstanceID.NotFound: The instance ID 'i-0f38265656a7c6ba9' does not exist\n\tstatus code: 400, request id: 2c373c57-9754-4205-931f-c517d30544c3\n\n" ], "provider": "urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0::1134e720-e72f-496e-a8f6-691b2002752f", "propertyDependencies": { "ami": null, "instanceType": null, "keyName": [ "urn:pulumi:dev::video_search::aws:ec2/keyPair:KeyPair::keypair" ], "subnetId": [ "urn:pulumi:dev::video_search::aws:ec2/subnet:Subnet::subnet" ], "tags": null, "userData": null, "vpcSecurityGroupIds": [ "urn:pulumi:dev::video_search::aws:ec2/securityGroup:SecurityGroup::secGroup" ] }, "additionalSecretOutputs": [ "tagsAll" ], "created": "2023-10-17T10:08:09.079176Z", "modified": "2023-10-18T16:33:43.837455Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_aws/ec2/instance.py#2180" }, { "urn": "urn:pulumi:dev::video_search::pulumi:providers:gcp::default", "custom": true, "id": "02e6e1f1-51c9-41e3-beac-5b3964b0dc14", "type": "pulumi:providers:gcp", "inputs": { "project": "pulumi-development", "version": "6.67.0" }, "outputs": { "project": "pulumi-development", "version": "6.67.0" }, "created": "2023-10-17T12:32:46.151309Z", "modified": "2023-10-17T12:32:46.151309Z" }, { "urn": "urn:pulumi:dev::video_search::pulumi:providers:gcp::default_6_67_0", "custom": true, "id": "ed53ac15-66de-4d12-98f8-3ba5cc4a011a", "type": "pulumi:providers:gcp", "inputs": { "project": "pulumi-development", "version": "6.67.0" }, "outputs": { "project": "pulumi-development", "version": "6.67.0" }, "created": "2023-10-17T12:32:47.681381Z", "modified": "2023-10-17T12:32:47.681381Z" }, { "urn": "urn:pulumi:dev::video_search::gcp:compute/globalAddress:GlobalAddress::vertex-range", "custom": true, "id": "projects/pulumi-development/global/addresses/vertex-range-c930f6e", "type": "gcp:compute/globalAddress:GlobalAddress", "inputs": { "__defaults": [ "name" ], "addressType": "INTERNAL", "name": "vertex-range-c930f6e", "network": "projects/pulumi-development/global/networks/default", "prefixLength": 24, "purpose": "VPC_PEERING" }, "outputs": { "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":1200000000000,\"delete\":1200000000000,\"update\":1200000000000}}", "address": "10.66.115.0", "addressType": "INTERNAL", "creationTimestamp": "2023-10-17T05:36:13.048-07:00", "description": "", "id": "projects/pulumi-development/global/addresses/vertex-range-c930f6e", "ipVersion": "", "labelFingerprint": "42WmSpB8rSM=", "labels": {}, "name": "vertex-range-c930f6e", "network": "https://www.googleapis.com/compute/v1/projects/pulumi-development/global/networks/default", "prefixLength": 24, "project": "pulumi-development", "purpose": "VPC_PEERING", "selfLink": "https://www.googleapis.com/compute/v1/projects/pulumi-development/global/addresses/vertex-range-c930f6e" }, "parent": "urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev", "provider": "urn:pulumi:dev::video_search::pulumi:providers:gcp::default_6_67_0::ed53ac15-66de-4d12-98f8-3ba5cc4a011a", "propertyDependencies": { "addressType": null, "network": null, "prefixLength": null, "purpose": null }, "created": "2023-10-17T12:36:24.205594Z", "modified": "2023-10-17T12:36:24.205594Z", "sourcePosition": "project:///venv/lib/python3.11/site-packages/pulumi_gcp/compute/global_address.py#670" } ] } } ```

The code which I am running pulumi up on is here.

Let me know if you need anything else.

Output of pulumi about

pulumi about
CLI
Version      3.89.0
Go Version   go1.21.1
Go Compiler  gc

Plugins
NAME    VERSION
aws     6.5.0
gcp     6.67.0
python  unknown

Host
OS       darwin
Version  13.5
Arch     arm64

This project is written in python: executable='/Users/vvm/.pyenv/shims/python3' version='3.11.6'

Current Stack: venelin-pulumi-corp/video_search/dev

TYPE                                                 URN
pulumi:providers:aws                                 urn:pulumi:dev::video_search::pulumi:providers:aws::default
pulumi:pulumi:Stack                                  urn:pulumi:dev::video_search::pulumi:pulumi:Stack::video_search-dev
pulumi:providers:aws                                 urn:pulumi:dev::video_search::pulumi:providers:aws::default_6_5_0
aws:ec2/vpc:Vpc                                      urn:pulumi:dev::video_search::aws:ec2/vpc:Vpc::vpc
aws:ec2/keyPair:KeyPair                              urn:pulumi:dev::video_search::aws:ec2/keyPair:KeyPair::keypair
aws:ec2/internetGateway:InternetGateway              urn:pulumi:dev::video_search::aws:ec2/internetGateway:InternetGateway::gateway
aws:ec2/subnet:Subnet                                urn:pulumi:dev::video_search::aws:ec2/subnet:Subnet::subnet
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:dev::video_search::aws:ec2/securityGroup:SecurityGroup::secGroup
aws:ec2/routeTable:RouteTable                        urn:pulumi:dev::video_search::aws:ec2/routeTable:RouteTable::routeTable
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:dev::video_search::aws:ec2/routeTableAssociation:RouteTableAssociation::routeTableAssociation
aws:ec2/instance:Instance                            urn:pulumi:dev::video_search::aws:ec2/instance:Instance::server
pulumi:providers:gcp                                 urn:pulumi:dev::video_search::pulumi:providers:gcp::default
pulumi:providers:gcp                                 urn:pulumi:dev::video_search::pulumi:providers:gcp::default_6_67_0
gcp:compute/globalAddress:GlobalAddress              urn:pulumi:dev::video_search::gcp:compute/globalAddress:GlobalAddress::vertex-range


Found no pending operations associated with dev

Backend
Name           pulumi.com
URL            https://app.pulumi.com/venelin-pulumi-corp
User           venelin-pulumi-corp
Organizations  venelin-pulumi-corp, pulumi
Token type     personal

Pulumi locates its logs in /var/folders/82/nqnqw81s1h56l5nv940f9mq00000gn/T/ by default
warning: Failed to get information about the Pulumi program's dependencies: illegal semver returned by language host: [email protected]: Invalid character(s) found in patch number "0.post2"

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@VenelinMartinov VenelinMartinov added kind/bug Some behavior is incorrect or out of spec impact/panic This bug represents a panic or unexpected crash needs-triage Needs attention from the triage team labels Oct 19, 2023
@t0yv0 t0yv0 added p1 A bug severe enough to be the next item assigned to an engineer and removed needs-triage Needs attention from the triage team labels Oct 19, 2023
@danielrbradley
Copy link
Member

Outcomes from pairing on debuggins with @VenelinMartinov

When calling d.GetRawPlan() in intercept.go, there's no d.diff.RawPlan and no d.state.RawPlan and so a cty.NullVal is returned. The panic is when we call GetAttr("tags_all") on the nil value. The gocty GetAttr code panics when calling val.v.(map[string]interface{})[name],` because v is nil.

Open questions:

  • Does this code break when other properties change (is it specific to userData)?
  • Did this work in an earlier version (what's changed in the interceptor or elsewhere)?

This looks suspliciously related to tags_all cc: @t0yv0

@t0yv0
Copy link
Member

t0yv0 commented Nov 3, 2023

Yeah. We have some temporarily placeholders for GetRawPlan instead of the correct impl. Any more context? A repro would be invaluable still.

@t0yv0
Copy link
Member

t0yv0 commented Nov 3, 2023

We could patch provider.tagsResourceInterceptor to be tolerant of nils but I wonder if there's an opportunity instead to get a little closer to TF behavior with populating RawPlan.

@VenelinMartinov
Copy link
Contributor Author

Here's steps for reproing the issue:

  1. pulumi new the python-aws-vm template
  2. manually terminate the ec2 instance in the aws console.
  3. change the user data of the ec2 instance and pulumi up
  4. observe the following error:
aws:ec2:Instance (server):
    error: 1 error occurred:
    	* updating urn:pulumi:dev::aws_panic2::aws:ec2/instance:Instance::server: 1 error occurred:
    	* updating EC2 Instance (i-05346c603c785698d) user data: stopping EC2 Instance: IncorrectInstanceState: This instance 'i-05346c603c785698d' is not in a state from which it can be stopped.
    	status code: 400, request id: 308e93e4-552e-489b-8285-ee3dd1e41300
  1. pulumi up again.
  2. Observe the panic.

@t0yv0
Copy link
Member

t0yv0 commented Nov 6, 2023

Excellent, thank you!

@t0yv0
Copy link
Member

t0yv0 commented Nov 13, 2023

I can reproduce this as described on my machine. This is very interesting.

There is a bridge issue somewhere around simpleDiffViaPlanState that gets activated when computing the Update plan for the Instance here. We try to emulate TF CLI by computing the Plan via:

	proposedNewStateVal, err := proposedNew(res, priorStateVal, rawConfigVal)

The plan is non-nil. However it becomes nil after going through the SimpleDiff black box:

state.RawPlan.IsNull() // false
proposedNewStateVal.IsNull() // false
diff, err := res.SimpleDiff(ctx, state, planned, meta)
diff.RawPlan.IsNull() // true

This trips up the expectations of the tags interceptor that is definitely expecting non-nil plans. We can quick-fix by adding a patch to the tags interceptor to introduce a nil check, but the above feels like bad bridge behavior. I'd like to step through this in a debugger quick.

@t0yv0
Copy link
Member

t0yv0 commented Nov 14, 2023

This seems to be a discrepancy between how bridged providers and TF do planning per pulumi/pulumi-terraform-bridge#1505 - a very special case of the above.

The bridge calls res.SimpleDiff which is allowed to return terraform.NewInstanceDiff(), nil.

In the case of TF reference implementation here:

https://github.com/hashicorp/terraform-plugin-sdk/blob/28e631776d97f0a5a5942b3524814addbef90875/helper/schema/grpc_provider.go#L797

There is code that compensates for an empty diff and propagates RawPlan from PlanResourceChange to ApplyResourceChange, so that subsequently res.Apply has access to it. Therefore the provider safely assumes RawPlan is accessible, but in the Pulumi case it is nil and causes a panic.

t0yv0 added a commit to pulumi/pulumi-terraform-bridge that referenced this issue Nov 15, 2023
Making a surgical change to avoid nil pointer panics on accessing
RawPlan, coming from the AWS panic in
pulumi/pulumi-aws#2904

I have verified that this resolves 2904.

---------

Co-authored-by: Ian Wahbe <[email protected]>
t0yv0 added a commit that referenced this issue Nov 15, 2023
[Upstream v5.25.0 release
notes](https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.25.0)

Fixes #2983, #2904, #2971, #2900

- [x] Rebuild eks.Cluster patches; upstream moved to AWS SDK v2 for Go,
patches needed updates as well
- [x] Fix pulumi/pulumi-terraform-bridge#1523
in the bridge
- [x] Update bridge to include
pulumi/pulumi-terraform-bridge#1521 and
pulumi/pulumi-terraform-bridge#1520 fixes
affecting P1s in pulumi-aws
- [x] Build a Pulumi test for EKS Cluster add-on removal -> turns out
the property is a no-op, not needed
@t0yv0
Copy link
Member

t0yv0 commented Nov 15, 2023

Fixed in v6.9.0

@t0yv0 t0yv0 added the resolution/fixed This issue was fixed label Nov 15, 2023
@t0yv0 t0yv0 closed this as completed Nov 15, 2023
@t0yv0 t0yv0 added the bug/diff kind/bug related to Pulumi generating wrong diffs on preview or up. label Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug/diff kind/bug related to Pulumi generating wrong diffs on preview or up. impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Projects
None yet
Development

No branches or pull requests

3 participants