Network interfaces and an instance-level security groups may not be specified on the same request

[t0yv0]

What happened?

Customer stack fails with the following error in Pulumi:

Network interfaces and an instance-level security groups may not be specified on the same request

But the same command succeeds through AWS CLI

aws ec2 run-instances --launch-template LaunchTemplateId=lt-0c......,Version=30 --subnet-id subnet-08....

Expected Behavior

Pulumi on par with AWS CLI here.

Steps to reproduce

We were not able to access the original repro but here is a synthetic repro that reproduces the same error message. The key is having LaunchTemplate specifying security groups and Instance specifying SubnetId.

package main

import (
	"fmt"

	"github.com/pulumi/pulumi-aws/sdk/v5/go/aws"
	"github.com/pulumi/pulumi-aws/sdk/v5/go/aws/ec2"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	p "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {

	p.Run(func(ctx *p.Context) error {
		awsProvider, err := aws.NewProvider(ctx, "aws", &aws.ProviderArgs{
			Profile: p.String("devsandbox"),
			Region:  p.String("us-west-2"),
		})
		if err != nil {
			return fmt.Errorf("provider: %w", err)
		}

		vpc, err := ec2.NewVpc(ctx, "main", &ec2.VpcArgs{
			CidrBlock: pulumi.String("10.0.0.0/16"),
		}, p.Provider(awsProvider))
		if err != nil {
			return err
		}

		subnet, err := ec2.NewSubnet(ctx, "main", &ec2.SubnetArgs{
			VpcId:     vpc.ID(),
			CidrBlock: pulumi.String("10.0.1.0/24"),
		}, p.Provider(awsProvider))
		if err != nil {
			return err
		}

		sg, err := ec2.NewSecurityGroup(ctx, "allowTls", &ec2.SecurityGroupArgs{
			VpcId: vpc.ID(),
			Egress: ec2.SecurityGroupEgressArray{
				&ec2.SecurityGroupEgressArgs{
					FromPort: pulumi.Int(0),
					ToPort:   pulumi.Int(0),
					Protocol: pulumi.String("-1"),
					CidrBlocks: pulumi.StringArray{
						pulumi.String("0.0.0.0/0"),
					},
					Ipv6CidrBlocks: pulumi.StringArray{
						pulumi.String("::/0"),
					},
				},
			},
		}, p.Provider(awsProvider))
		if err != nil {
			return err
		}

		ami, err := getSomeAmi(ctx, awsProvider)
		if err != nil {
			return err
		}

		launchTempl, err := ec2.NewLaunchTemplate(ctx, "exampleLaunchTemplate", &ec2.LaunchTemplateArgs{
			ImageId:            pulumi.String(ami),
			InstanceType:       pulumi.String("t2.micro"),
			SecurityGroupNames: pulumi.StringArray{sg.Name},
		}, p.Provider(awsProvider))
		if err != nil {
			return err
		}

		_, err = ec2.NewInstance(ctx, "debug-worker", &ec2.InstanceArgs{
			SubnetId:     subnet.ID(),
			InstanceType: pulumi.String("t2.micro"),
			LaunchTemplate: &ec2.InstanceLaunchTemplateArgs{
				Id: launchTempl.ID(),
				Version: launchTempl.LatestVersion.ApplyT(func(x int) *string {
					res := fmt.Sprintf("%d", x)
					return &res
				}).(pulumi.StringPtrOutput),
			},
		}, p.Provider(awsProvider))
		return err
	})
}

func getSomeAmi(ctx *pulumi.Context, awsProvider *aws.Provider) (string, error) {
	mostRecent := true
	ami, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
		Filters: []ec2.GetAmiFilter{
			{
				Name:   "name",
				Values: []string{"amzn-ami-hvm-*-x86_64-ebs"},
			},
		},
		Owners:     []string{"137112412989"},
		MostRecent: &mostRecent,
	}, p.Provider(awsProvider))
	if err != nil {
		return "", err
	}
	return ami.Id, nil
}

Output of pulumi about

CLI          
Version      3.64.0
Go Version   go1.20.3
Go Compiler  gc

Plugins
NAME  VERSION
aws   5.33.0
go    unknown

Host     
OS       darwin
Version  13.1
Arch     x86_64

This project is written in go: executable='/Users/t0yv0/.nix-profile/bin/go' version='go version go1.20.1 darwin/amd64'

Current Stack: t0yv0/repro-bug/dev

Found no resources associated with dev

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/t0yv0
User           t0yv0
Organizations  t0yv0, pulumi

Dependencies:
NAME                                 VERSION
github.com/pulumi/pulumi-aws/sdk/v5  5.33.0
github.com/pulumi/pulumi/sdk/v3      3.60.1

Pulumi locates its logs in /var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

I can work around the error if I change the launch template to specify NetworkInterfaces instead of SecurityGroupNames and associate the security group to the NetworkInterface:

		launchTempl, err := ec2.NewLaunchTemplate(ctx, "exampleLaunchTemplate", &ec2.LaunchTemplateArgs{
			ImageId:      pulumi.String(ami),
			InstanceType: pulumi.String("t2.micro"),
			//SecurityGroupNames: pulumi.StringArray{sg.Name},
			NetworkInterfaces: ec2.LaunchTemplateNetworkInterfaceArray{
				ec2.LaunchTemplateNetworkInterfaceArgs{
					SubnetId: subnet.ID(),
					SecurityGroups: p.StringArray{
						sg.ID(),
					},
				},
			},
		}, p.Provider(awsProvider))
		if err != nil {
			return err
		}

Also removing subnet association from the instance and to the launch template network interface:

		_, err = ec2.NewInstance(ctx, "debug-worker", &ec2.InstanceArgs{
			//SubnetId:     subnet.ID(),
			InstanceType: pulumi.String("t2.micro"),
			LaunchTemplate: &ec2.InstanceLaunchTemplateArgs{
				Id: launchTempl.ID(),
				Version: launchTempl.LatestVersion.ApplyT(func(x int) *string {
					res := fmt.Sprintf("%d", x)
					return &res
				}).(pulumi.StringPtrOutput),
			},
		}, p.Provider(awsProvider))

[t0yv0]

Also perhaps related: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#security_group_names and https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#vpc_security_group_ids

[t0yv0]

Perhaps what's happening here is that this configuration of LaunchTemplate SecurityGroups is not compatible with NewInstance, but the surfaced error is not very direct about how to fix that.

[mikhailshilkov]

@t0yv0 I can see you marked it as awaiting-feedback - are we waiting for some input here? Is the issue still relevant to the customer? Reading the symptoms, it sounds like an upstream issue to me, WDYT?

[t0yv0]

I've heard indirectly (@phillipedwards) that the workaround was acceptable. I think we can close.

[pulumi-bot]

Cannot close issue without required labels: resolution/