Panic provisioning aws:ec2:VpcEndpoint on AWS V6.0 #2658

ringods · 2023-08-01T14:42:06Z

What happened?

Customer experienced a panic after upgrading to Pulumi CLI v3.76.1:

aws:ec2:VpcEndpoint (secretsManagerVPCEdpoint):

    error: error reading from server: EOF

  pulumi:pulumi:Stack (<customer-stack-name>):

    warning: provider config warning: skip_metadata_api_check: the use of values other than "true" and "false" is deprecated and will be removed in a future version of the provider

    error: update failed

    panic: interface conversion: interface {} is nil, not map[string]interface {}

    goroutine 83 [running]:
    github.com/hashicorp/go-cty/cty.Value.GetAttr({{{0x111302230?, 0x140083ce808?}}, {0x0?, 0x0?}}, {0x10d424969, 0x8})
       [/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/cty/value_ops.go:711](mailto:/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/cty/value_ops.go:711) +0x268
    github.com/hashicorp/terraform-provider-aws/internal/provider.tagsInterceptor.run({0x14000685700?, 0x1111e5078?, 0x1111e5070?}, {0x111300ac8, 0x14008417170}, {0x11130eb20, 0x14001085900}, {0x11117d5a0?, 0x14002b04750?}, 0x1, ...)
       /home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:250 +0xea4
    github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0x0?, {0x11117d5a0?, 0x14002b04750?})
       /home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:100 +0x114
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0x111300a58?, {0x111300a58?, 0x140000b6080?}, 0xd?, {0x11117d5a0?, 0x14002b04750?})
       [/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:767](mailto:/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:767) +0x64
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0x1400119fdc0, {0x111300a58, 0x140000b6080}, 0x1400607a4e0, 0x14001085400, {0x11117d5a0, 0x14002b04750})
       [/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:879](mailto:/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:879) +0x6b0
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0x140008cc2a0?, {0x14000638a30?, 0x140083d3670?, 0x0?}}, {0x10d467442, 0x10}, {0x111302700?, 0x140082c9ea8}, {0x11130f888?, 0x14001085400})
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:100](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:100) +0x160
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0x14003f4a580, {0x111300ac8?, 0x140082e3ce0?}, 0x14007f69a70)
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:929](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:929) +0x688
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Update.func1({0x111327390?, 0x14003f4a580?})
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:356](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:356) +0x3c
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.resourceMethod[...](0x14001729860?, 0x1400820d778, 0x1400820d758?)
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:303](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:303) +0xc4
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Update(0x1400820d7b8?, {0x111300ac8?, 0x140082e3ce0?}, 0x10ef4d440?)
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:355](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:355) +0x5c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0x111300ac8, 0x140082e3ce0}, {0x110dea180?, 0x14007f69a70})
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:609](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:609) +0x74
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x111300ac8, 0x140082e34d0}, {0x110dea180, 0x14007f69a70}, 0x140082dd8c0, 0x140082c99e0)
       [/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57](mailto:/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57) +0x308
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0x111008900?, 0x14001729860}, {0x111300ac8, 0x140082e34d0}, 0x140082cfb90, 0x14003f0e6c0)
       [/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:611](mailto:/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:611) +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0x140000b23c0, {0x1113107e0, 0x140036a21a0}, 0x140082ed680, 0x140058d6690, 0x1197c23f0, 0x0)
       [/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1337](mailto:/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1337) +0xc90
    google.golang.org/grpc.(*Server).handleStream(0x140000b23c0, {0x1113107e0, 0x140036a21a0}, 0x140082ed680, 0x0)
       [/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1714](mailto:/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1714) +0x82c
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
       [/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:959](mailto:/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:959) +0x84
    created by google.golang.org/grpc.(*Server).serveStreams.func1
       [/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:957](mailto:/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:957) +0x16c

Expected Behavior

No panic.

Steps to reproduce

Code snippet:

new VpcEndpoint("secretsManagerVPCEdpoint", VpcEndpointArgs.builder()
  .dnsOptions(VpcEndpointDnsOptionsArgs.builder()
    .dnsRecordIpType("ipv4")
    .build())
  .ipAddressType("ipv4")
  .policy(readJson("secretsManagerVPCEndpointPolicy"))
  .privateDnsEnabled(true)
  .securityGroupIds(Output.all(asmVpcEndpointSecurityGroup.id()))
  .serviceName("com.amazonaws.eu-west-1.secretsmanager")
  .subnetIds(getPrivateSubnetIds(defaultVpc))
  .tags(Map.of("Name", "secrets manager endpoint"))
  .vpcEndpointType("Interface")
  .vpcId(defaultVpcId)
  .build(), CustomResourceOptions.builder()
    .protect(true)
    .build());

Output of `pulumi about`

CLI
Version 3.76.1
Go Version go1.20.6
Go Compiler gc

Plugins
NAME VERSION
aws 6.0.0-alpha.5+8163083e
java unknown

Host
OS darwin
Version 13.5
Arch arm64

This project is written in java: executable='/Users/<user>/java/jdk-11.0.2.jdk/Contents/Home/bin/java' version='openjdk 11.0.2 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)' javac='11.0.2' maven='Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)' java='/Users/<user>/java/jdk-11.0.2.jdk/Contents/Home/bin/java'

Backend
Name pulumi.com
URL https://app.pulumi.com/<name>
User <name>
Organizations <org>

No dependencies found

Pulumi locates its logs in /var/folders/_6/_f9gssfj5nq97s5gx50cys098s9s5z/T/ by default
warning: Failed to get information about the current stack: No current stack

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

Frassle · 2023-08-01T15:02:28Z

That stacktrace looks like a pulumi-aws issue rather than the CLI?

ringods · 2023-08-01T15:19:00Z

@Frassle customer responded with pulumi about (updated above). They are using AWS 6.0.0-alpha.5+8163083e

dixler · 2023-08-01T15:41:23Z

Yeah. This looks like a provider panic, I'll transfer it to pulumi-aws.

It's interesting that it's using java.

t0yv0 · 2023-08-01T16:07:19Z

It's interesting that it's using java.

Unfortunately, Java has a limitation that it automatically upgrades to -alpha releases.

The workaround is to pin a latest production release of pulumi-aws, v5.42.0

guineveresaenger · 2023-08-02T00:19:43Z

Internal discussion on Slack

t0yv0 · 2023-08-02T20:08:58Z

I took at stab in reproducing this but I'm having trouble reproducing.

The program I came up with is

package myproject;

import java.util.Map;
import com.pulumi.Pulumi;
import com.pulumi.resources.CustomResourceOptions;
import com.pulumi.core.Output;
import com.pulumi.aws.s3.Bucket;
import com.pulumi.aws.ec2.Vpc;
import com.pulumi.aws.ec2.VpcArgs;
import com.pulumi.aws.ec2.VpcEndpoint;
import com.pulumi.aws.ec2.VpcEndpointArgs;
import com.pulumi.aws.ec2.inputs.VpcEndpointDnsOptionsArgs;

public class App {
    public static void main(String[] args) {
        Pulumi.run(ctx -> {

           var defaultVpc = new Vpc("mainvpc", VpcArgs.builder()
                                   .cidrBlock("10.0.0.0/16")
                                   .enableDnsHostnames(true)
                                   .enableDnsSupport(true)
                                   .build());

           var defaultVpcId = defaultVpc.getId();

           var vpce = new VpcEndpoint("secretsManagerVPCEdpoint", VpcEndpointArgs.builder()
                                .dnsOptions(VpcEndpointDnsOptionsArgs.builder()
                                            .dnsRecordIpType("ipv4")
                                            .build())
                                .ipAddressType("ipv4")
                                 // .policy("{}")
                                .privateDnsEnabled(true)
                                 // .securityGroupIds(Output.all(asmVpcEndpointSecurityGroup.id()))
                                .serviceName("com.amazonaws.us-east-1.secretsmanager")
                                 // .subnetIds(getPrivateSubnetIds(defaultVpc))
                                      .tags(Map.of("Name", "secrets manager endpoint"))
                                .vpcEndpointType("Interface")
                                .vpcId(defaultVpcId)
                                .build(), CustomResourceOptions.builder()
                                      // .protect(true)
                                .build());
        });
    }
}

I tried it under both the last released version and the cited version:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.pulumi</groupId>
    <artifactId>regress-2539</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <encoding>UTF-8</encoding>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <maven.compiler.release>11</maven.compiler.release>
        <mainClass>myproject.App</mainClass>
        <mainArgs/>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.pulumi</groupId>
            <artifactId>pulumi</artifactId>
            <version>(,1.0]</version>
        </dependency>
        <dependency>
            <groupId>com.pulumi</groupId>
            <artifactId>aws</artifactId>
            <version>5.42.0</version>
            <!-- <version>6.0.0-alpha.5+8163083e</version> -->
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.2</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>${mainClass}</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>${mainClass}</mainClass>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-my-jar-with-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <mainClass>${mainClass}</mainClass>
                    <commandlineArgs>${mainArgs}</commandlineArgs>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-wrapper-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <mavenVersion>3.8.5</mavenVersion>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

I tried adding/removing tags. In all the combinations I'm not able to hit the panic. It seems from the stack trace that this is executing Update part of the plan, so I tried updates, but no luck so far. I tried an Upgrade test from 5.* to 6.* on a provisioned program, also no luck.

t0yv0 · 2023-08-02T20:10:12Z

One set of possibilities I did not test is perhaps default (provider-level) tags are in play somehow.

Currently unable to set them to test out due to
#2663

t0yv0 · 2023-08-03T13:39:22Z

Looks like #2663 was invalid, it's still possible to try defaultTags here. Unfortunately I need to pivot for the rest of the week, back to this on Monday.

t0yv0 · 2023-08-07T15:32:35Z

Deeper investigation reveals this is a duplicate of pulumi/pulumi-terraform-bridge#1033 that was fixed after the issue was discovered.

Where the panic originates:

upstream/internal/provider/intercept.go:250:
    if d.GetRawPlan().GetAttr("tags_all").IsWhollyKnown()

Recent fix to not panic on GetRawPlan:
pulumi/pulumi-terraform-bridge#1033

Merged to AWS in:
#2494 (branch t0yv0/bridge-v3.45.2)

Affected commit is on v3.54.1:

pulumi-aws/provider/go.mod

Line 11 in 8163083

github.com/pulumi/pulumi-terraform-bridge/v3 v3.54.1

ringods added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Aug 1, 2023

mnlumi added the customer/feedback Feedback from customers label Aug 1, 2023

joeduffy added the p1 A bug severe enough to be the next item assigned to an engineer label Aug 1, 2023

dixler transferred this issue from pulumi/pulumi Aug 1, 2023

t0yv0 added 6.0 and removed p1 A bug severe enough to be the next item assigned to an engineer labels Aug 1, 2023

t0yv0 changed the title ~~Panic when updating Pulumi CLI to v3.76.1~~ Panic provisioning aws:ec2:VpcEndpoint on AWS V6.0 Aug 1, 2023

t0yv0 mentioned this issue Aug 1, 2023

Dev builds published to Maven central causing automatic upgrades to -alpha pulumi/ci-mgmt#1107

Open

mnlumi added the impact/panic This bug represents a panic or unexpected crash label Aug 1, 2023

guineveresaenger removed the needs-triage Needs attention from the triage team label Aug 2, 2023

mnlumi mentioned this issue Aug 2, 2023

[Epic] Pulumi AWS 6.0 #2539

Closed

34 tasks

t0yv0 self-assigned this Aug 2, 2023

t0yv0 added this to the 0.92 milestone Aug 2, 2023

mnlumi mentioned this issue Aug 2, 2023

Prevent java template from using 6.0 alpha versions pulumi/templates#624

Merged

t0yv0 added the needs-repro Needs repro steps before it can be triaged or fixed label Aug 2, 2023

t0yv0 assigned t0yv0 and unassigned t0yv0 Aug 3, 2023

t0yv0 added resolution/duplicate This issue is a duplicate of another issue and removed needs-repro Needs repro steps before it can be triaged or fixed labels Aug 7, 2023

t0yv0 closed this as completed Aug 7, 2023

mikhailshilkov added resolution/fixed This issue was fixed and removed resolution/duplicate This issue is a duplicate of another issue labels May 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Panic provisioning aws:ec2:VpcEndpoint on AWS V6.0 #2658

Panic provisioning aws:ec2:VpcEndpoint on AWS V6.0 #2658

ringods commented Aug 1, 2023 •

edited

Loading

Frassle commented Aug 1, 2023

ringods commented Aug 1, 2023

dixler commented Aug 1, 2023 •

edited

Loading

t0yv0 commented Aug 1, 2023

guineveresaenger commented Aug 2, 2023

t0yv0 commented Aug 2, 2023

t0yv0 commented Aug 2, 2023

t0yv0 commented Aug 3, 2023

t0yv0 commented Aug 7, 2023

Panic provisioning aws:ec2:VpcEndpoint on AWS V6.0 #2658

Panic provisioning aws:ec2:VpcEndpoint on AWS V6.0 #2658

Comments

ringods commented Aug 1, 2023 • edited Loading

What happened?

Expected Behavior

Steps to reproduce

Output of pulumi about

Additional context

Contributing

Frassle commented Aug 1, 2023

ringods commented Aug 1, 2023

dixler commented Aug 1, 2023 • edited Loading

t0yv0 commented Aug 1, 2023

guineveresaenger commented Aug 2, 2023

t0yv0 commented Aug 2, 2023

t0yv0 commented Aug 2, 2023

t0yv0 commented Aug 3, 2023

t0yv0 commented Aug 7, 2023

ringods commented Aug 1, 2023 •

edited

Loading

Output of `pulumi about`

dixler commented Aug 1, 2023 •

edited

Loading