AWS: SNS Topic subscription ignores filter_policy_scope

[prudnikov]

What happened?

When creating sns.TopicSubscription there is filter_policy_scope property which have 2 options: "MessageAttributes" is default, and "MessageBody". No matter which value is used, it will always create a subscription with "Message Attributes" selected.

Expected Behavior

When filter_policy_scope="MessageBody" it should create a subscription with "Message Body" filter policy.

Steps to reproduce

    sns.TopicSubscription(
        f"dev-dataplatform-alerting-xxxx",
        sns.TopicSubscriptionArgs(
            topic=alerting_topic.get_arn(),
            protocol=target.protocol,
            endpoint=target.endpoint,
            endpoint_auto_confirms=target.endpoint_auto_confirms,
            filter_policy=filter_policy,
            filter_policy_scope="MessageBody",
        ),
    )

Output of pulumi about

❯ pulumi about                                 
CLI          
Version      3.78.1
Go Version   go1.20.7
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.42.0
docker  4.3.0
python  unknown

Host     
OS       darwin
Version  13.5
Arch     arm64

This project is written in python: executable='/Users/vprudnikov/Code/PROJECT/.venv/bin/python3' version='3.11.4
'

Current Stack: PROJECT

TYPE                                                URN
pulumi:pulumi:Stack                                 urn:pulumi:PROJECT::PLATFORM::pulumi:pulumi:Stack::PLATFORM-PROJECT
pulumi:providers:aws                                urn:pulumi:PROJECT::PLATFORM::pulumi:providers:aws::default
pcty:PROJECT:sns                                    urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns::PROJECT
pcty:PROJECT:default-catch-all-queue                urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns$pcty:PROJECT:default-catch-all-queue::dev-dataplatform-default-catch-all-queue
pulumi:providers:aws                                urn:pulumi:PROJECT::PLATFORM::pulumi:providers:aws::default_5_42_0
aws:sns/topic:Topic                                 urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns$aws:sns/topic:Topic::PROJECT
pulumi:providers:pulumi                             urn:pulumi:PROJECT::PLATFORM::pulumi:providers:pulumi::default
aws:sqs/queue:Queue                                 urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns$pcty:PROJECT:default-catch-all-queue$aws:sqs/queue:Queue::PROJECT-default-catch-all-queue
aws:sns/topicSubscription:TopicSubscription         urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns$pcty:PROJECT:default-catch-all-queue$aws:sns/topicSubscription:TopicSubscription::PROJECT-default-catch-all-queue-subscription
aws:sqs/queuePolicy:QueuePolicy                     urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns$pcty:PROJECT:default-catch-all-queue$aws:sqs/queuePolicy:QueuePolicy::PROJECT-default-catch-all-queue-policy
pulumi:pulumi:StackReference                        urn:pulumi:PROJECT::PLATFORM::pulumi:pulumi:StackReference::dev-dataplatform-payroll
aws:sns/topicSubscription:TopicSubscription         urn:pulumi:PROJECT::PLATFORM::pcty:PROJECT:sns$aws:sns/topicSubscription:TopicSubscription::PROJECT-my-email


Found no pending operations associated with PROJECT

Backend        
Name           vprudnikov-mac
URL            s3://COMPANY-dev-pulumi-state
User           vprudnikov
Organizations  

Dependencies:
NAME             VERSION
alerting         1.0.0
black            23.7.0
flake8-builtins  2.1.0
flake8-pie       0.16.0
pip              23.2.1
pre-commit       3.3.3
pylint           2.17.4
pytest-cov       4.1.0
pytest-mock      3.11.1
radon            6.0.1
requests-mock    1.11.0

Pulumi locates its logs in /var/folders/l8/mpdm5ht55rq6hh1dgb09vflm0000gq/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[Zaid-Ajaj]

Hi there @prudnikov thank you for filing the issue and for the detailed report! This seems to be bug in the AWS provider, I will be moving the issue to pulumi-aws where the providers team can look into it

[danielrbradley]

I've managed to reproduce this filling the extra details for the program:

import * as aws from "@pulumi/aws";

const topic = new aws.sns.Topic("dev-dataplatform-alerting-xxxx");

const queue = new aws.sqs.Queue("dev-dataplatform-alerting-xxxx");

export const sub = new aws.sns.TopicSubscription(
  "dev-dataplatform-alerting-xxxx",
  {
    topic: topic.arn,
    protocol: "sqs",
    endpoint: queue.arn,
    endpointAutoConfirms: true,
    filterPolicy: '{"foo":["bar"]}',
    filterPolicyScope: "MessageBody",
  }
);

Manual changes

When editing the resource directly in the console then performing a refresh we see the following diff:

 ~   └─ aws:sns:TopicSubscription  dev-dataplatform-alerting-xxxx  update     [diff: ~filterPolicyScope]


Resources:
    ~ 1 to update
    3 unchanged

Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::scratch::pulumi:pulumi:Stack::scratch-dev]
    ~ aws:sns/topicSubscription:TopicSubscription: (update)
        [id=arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d]
        [urn=urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx]
        [provider=urn:pulumi:dev::scratch::pulumi:providers:aws::default_5_42_0::57e6efc2-2886-4a07-807b-395b1dcdb17c]
      ~ filterPolicyScope: "MessageBody" => "MessageBody"

Preliminary Investigation

Relevant lines from the verbose log:

I0823 16:15:14.665870   62313 log.go:81] Unmarshaling property for RPC[ResourceMonitor.RegisterResource(aws:sns/topicSubscription:TopicSubscription,dev-dataplatform-alerting-xxxx)]: filterPolicyScope={MessageBody}
...
I0823 16:15:14.666318   62313 log.go:81] Marshaling property for RPC[Provider[aws, 0x14000989bd0].Check(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx).news]: filterPolicyScope={MessageBody}
...
I0823 16:15:14.667525   62313 log.go:81] eventSink::Infoerr(<{%reset%}>I0823 16:15:14.667284   62375 log.go:81] Unmarshaling property for RPC[tf.Provider[aws].Check(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx/aws_sns_topic_subscription).news]: filterPolicyScope={MessageBody}
...
I0823 16:15:14.667659   62313 log.go:81] eventSink::Infoerr(<{%reset%}>I0823 16:15:14.667470   62375 schema.go:888] Terraform output filterPolicyScope = {MessageBody}
...
I0823 16:15:14.668316   62313 log.go:81] eventSink::Infoerr(<{%reset%}>I0823 16:15:14.667503   62375 log.go:81] Marshaling property for RPC[tf.Provider[aws].Check(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx/aws_sns_topic_subscription).inputs]: filterPolicyScope={MessageBody}
...
I0823 16:15:14.668455   62313 log.go:81] Unmarshaling property for RPC[Provider[aws, 0x14000989bd0].Check(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx).inputs]: filterPolicyScope={MessageBody}
...
I0823 16:15:14.668518   62313 log.go:81] Planner decided to create 'urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx' (inputs=map[__defaults:{[{confirmationTimeoutInMinutes} {rawMessageDelivery}]} confirmationTimeoutInMinutes:{1} endpoint:{arn:aws:sqs:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-78bd789} endpointAutoConfirms:{true} filterPolicy:{{"foo":["bar"]}} filterPolicyScope:{MessageBody} protocol:{sqs} rawMessageDelivery:{false} topic:{arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c}])
...
I0823 16:15:14.918461   62313 log.go:81] Marshaling property for RPC[Provider[aws, 0x14000989bd0].Create(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx).inputs]: filterPolicyScope={MessageBody}
...
I0823 16:15:14.921411   62313 log.go:81] eventSink::Infoerr(<{%reset%}>I0823 16:15:14.921355   62375 log.go:81] Unmarshaling property for RPC[tf.Provider[aws].Create(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx/aws_sns_topic_subscription).news]: filterPolicyScope={MessageBody}
...
I0823 16:15:15.280692   62313 log.go:81] eventSink::Infoerr(<{%reset%}>I0823 16:15:15.280581   62375 schema.go:888] Terraform output filterPolicyScope = {MessageAttributes}
...
I0823 16:15:15.281542   62313 log.go:81] Unmarshaling property for RPC[Provider[aws, 0x14000989bd0].Create(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx).outputs]: filterPolicyScope={MessageAttributes}
...
I0823 16:15:15.282208   62313 log.go:81] eventSink::Infoerr(<{%reset%}>I0823 16:15:15.280824   62375 log.go:81] Marshaling property for RPC[tf.Provider[aws].Create(urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx/aws_sns_topic_subscription).outs]: filterPolicyScope={MessageAttributes}
...
I0823 16:15:15.568312   62313 log.go:81] Marshaling property for RPC[ResourceMonitor.RegisterResource(aws:sns/topicSubscription:TopicSubscription,dev-dataplatform-alerting-xxxx)]: filterPolicyScope={MessageAttributes}
...
I0823 16:15:15.577276   62313 log.go:81] RegisterResource RPC finished: resource:dev-dataplatform-alerting-xxxx[aws:sns/topicSubscription:TopicSubscription]; err: null, resp: urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d,arn,,,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d,confirmationTimeoutInMinutes,,1,confirmationWasAuthenticated,,,,true,deliveryPolicy,,,,endpoint,,,arn:aws:sqs:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-78bd789,endpointAutoConfirms,,,,true,filterPolicy,,,{"foo":["bar"]},filterPolicyScope,,,MessageAttributes,id,,,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d,ownerId,,,REDACTED,pendingConfirmation,,,,false,protocol,,,sqs,rawMessageDelivery,,,,false,redrivePolicy,,,,subscriptionRoleArn,,,,topic,,,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c,,
I0823 16:15:15.577347   62313 log.go:81] eventSink::Debug(<{%reset%}>RegisterResource RPC finished: resource:dev-dataplatform-alerting-xxxx[aws:sns/topicSubscription:TopicSubscription]; err: null, resp: urn:pulumi:dev::scratch::aws:sns/topicSubscription:TopicSubscription::dev-dataplatform-alerting-xxxx,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d,arn,,,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d,confirmationTimeoutInMinutes,,1,confirmationWasAuthenticated,,,,true,deliveryPolicy,,,,endpoint,,,arn:aws:sqs:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-78bd789,endpointAutoConfirms,,,,true,filterPolicy,,,{"foo":["bar"]},filterPolicyScope,,,MessageAttributes,id,,,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c:3a2e89e7-1a5b-4230-8274-04c315d3c92d,ownerId,,,REDACTED,pendingConfirmation,,,,false,protocol,,,sqs,rawMessageDelivery,,,,false,redrivePolicy,,,,subscriptionRoleArn,,,,topic,,,arn:aws:sns:eu-west-2:REDACTED:dev-dataplatform-alerting-xxxx-e68c97c,,<{%reset%}>)

It therefore appears that we are passing the value correct as an input to the underlying provider, but the output value then doesn't match the input. This therefore seems likely to be an upstream issue - either in the TF provider logic, or the AWS service not honnoring the input provided.

There's this special logic for handling the field in the diff:

• https://github.com/pulumi/terraform-provider-aws/blob/225e17c8a683de1264a1e6d07e2ccc62852dead4/internal/service/sns/topic_subscription.go#L518-L551

[Sodki]

I think this is a duplicate of #2576

[Sodki]

I can no longer replicate using the v6 version of pulumi-aws.