update on aws.vpc.routeTable shows unnecessary diff/update

[phillipedwards]

As a developer, I want pulumi to correctly show whether or not my stack requires updates or not, so that I can correctly identify when I've made a change to my pulumi managed resources.

1. git clone https://gist.github.com/phillipedwards/f4604a2a5cbeb7761791f79ff2633859
2. pulumi up
3. re-run pulumi up
4. observe diff, showing update to route table

Expected: No updates made to aws resources on second execution of pulumi up
Actual: pulumi up results in a diff for the route table's CIDR block and gateway ID

[phillipedwards]

The diff related to gatewayId vs transitGatewayId appears to be a difference of opinion on types or polymorphism related. Eg- transitGateway is a type of gateway.

[saarw-opti]

Something new about it?
I get it also :|

---

Edit:

Change the from gateway_id var to nat_gateway_id and it will fix it.

[t0yv0]

Updating (repro-1574):
     Type                                     Name                       Status                        Info
 +   pulumi:pulumi:Stack                      aws-ts-subnets-repro-1574  **creating failed (51s)**     1 error
 +   ├─ aws:ec2:Vpc                           diff-vpc                   created (4s)
 +   ├─ aws:ec2transitgateway:TransitGateway  diff-example               created (47s)
 +   ├─ aws:ec2:Subnet                        diff-subnet                created (1s)
 +   ├─ aws:ec2transitgateway:VpcAttachment   diff-=attach               created (71s)
 +   └─ aws:ec2:RouteTable                    diff-route-table           **creating failed**           1 error

Diagnostics:
  pulumi:pulumi:Stack (aws-ts-subnets-repro-1574):
    error: update failed

  aws:ec2:RouteTable (diff-route-table):
    error: 1 error occurred:
        * creating urn:pulumi:repro-1574::aws-ts-subnets::aws:ec2/routeTable:RouteTable::diff-route-table: 1 error occurred:
        * error creating Route in Route Table (rtb-092ecf17a86f9364d) with destination (0.0.0.0/0): InvalidGatewayID.NotFound: The gateway ID 'tgw-0af86f9703986f3e2' does not exist
        status code: 400, request id: dc83ecc2-6787-4522-8757-cfe14c171a31

Outputs:
    gatewayId: "tgw-0af86f9703986f3e2"

Trying to reproduce this also noting that the first pulumi up completely fails for this program.

[t0yv0]

Couple more notes. Reproduces on 6.12.3 for me. Indeed fixing the program like this can workaround the issue:

new aws.ec2.RouteTable("diff-route-table", {
    vpcId: vpc.id,
    routes: [{
        cidrBlock: "0.0.0.0/0",
        // gatewayId: gateway.id
        transitGatewayId: gateway.id
    }]
});

[t0yv0]

I've translated this program to TF.

resource "aws_vpc" "myvpc" {
  cidr_block = "10.100.0.0/16"
}

resource "aws_ec2_transit_gateway" "tg" {
}

resource "aws_ec2_transit_gateway_vpc_attachment" "diffattach" {
  subnet_ids         = [aws_subnet.mysub.id]
  transit_gateway_id = aws_ec2_transit_gateway.tg.id
  vpc_id             = aws_vpc.myvpc.id
}

resource "aws_subnet" "mysub" {
  vpc_id     = aws_vpc.myvpc.id
  cidr_block = "10.100.1.0/24"
}

resource "aws_route_table" "example" {
  vpc_id = aws_vpc.myvpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_ec2_transit_gateway.tg.id
  }
}

It appears the same behavior is present in upstream 5.29.0 provider and TF CLI 1.6.3.

The first terraform apply fails with:

╷
│ Error: creating Route in Route Table (rtb-07e141eb97bd3b15c) with destination (0.0.0.0/0): InvalidGatewayID.NotFound: The gateway ID 'tgw-05f26a33b9e4b27bd' does not exist
│       status code: 400, request id: e9d50724-4877-45af-bafb-30920a42131a
│
│   with aws_route_table.example,
│   on infra.tf line 19, in resource "aws_route_table" "example":
│   19: resource "aws_route_table" "example" {
│
╵

Subsequent Terraform apply gets stuck in a permanent diff with:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_route_table.example will be updated in-place
  ~ resource "aws_route_table" "example" {
        id               = "rtb-0db9dc038995de7bb"
      ~ route            = [
          - {
              - carrier_gateway_id         = ""
              - cidr_block                 = "0.0.0.0/0"
              - core_network_arn           = ""
              - destination_prefix_list_id = ""
              - egress_only_gateway_id     = ""
              - gateway_id                 = ""
              - ipv6_cidr_block            = ""
              - local_gateway_id           = ""
              - nat_gateway_id             = ""
              - network_interface_id       = ""
              - transit_gateway_id         = "tgw-05f26a33b9e4b27bd"
              - vpc_endpoint_id            = ""
              - vpc_peering_connection_id  = ""
            },
          + {
              + carrier_gateway_id         = ""
              + cidr_block                 = "0.0.0.0/0"
              + core_network_arn           = ""
              + destination_prefix_list_id = ""
              + egress_only_gateway_id     = ""
              + gateway_id                 = "tgw-05f26a33b9e4b27bd"
              + ipv6_cidr_block            = ""
              + local_gateway_id           = ""
              + nat_gateway_id             = ""
              + network_interface_id       = ""
              + transit_gateway_id         = ""
              + vpc_endpoint_id            = ""
              + vpc_peering_connection_id  = ""
            },
        ]
        tags             = {}
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[t0yv0]

This appears somewhat documented https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table see note 2:

NOTE on gateway_id and nat_gateway_id:

The AWS API is very forgiving with these two attributes and the aws_route_table resource can be created with a NAT ID specified as a Gateway ID attribute. This will lead to a permanent diff between your configuration and statefile, as the API returns the correct parameters in the returned route table. If you're experiencing constant diffs in your aws_route_table resources, the first thing to check is whether or not you're specifying a NAT ID instead of a Gateway ID, or vice-versa.

It does not specifically mention transitGateway but I believe it behaves similarly here.

[t0yv0]

Looks like Pulumi documentation has also inherited this note: https://www.pulumi.com/registry/packages/aws/api-docs/ec2/routetable/

[t0yv0]

@iwahbe @phillipedwards based on the above investigation I suggest closing as "resolution/by-design". It will be relatively difficult for us to improve this behavior over what the upstream provider decided to be the by-design behavior. I'm open to other suggestions how to proceed here, perhaps we could enhance the Pulumi side documentation on the issue?

[iwahbe]

I agree with @t0yv0. The documentation correctly describes the observed behavior, and offers a path to resolve it. Upstream does the same thing.

@phillipedwards Does that work for you?

[t0yv0]

Closing. Please reopen if necessary.