Loosely Coupled Cross Stack References

[corrjo]

Looking for input:

Cloudformation exports work really well for consuming properties of long lived resources, like vpc or subnet ids, however when you have a cdk app with stacks and resources that have variable lifecycles, using exports can lead to a rough time with stacks or resources being locked because something is consuming its export leading to having to do something like https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references

How I handled this in CFN

Before using cdk, this was an issue that would pop up on occasion and this led me and my team to make broader use of nested stacks, and parameter store.

Nested stacks worked well enough, but it ended up requiring a lot of planning and overhead since passing values between nested stacks at the same level or more than 1 level deep becomes a bit convoluted, hence parameter store to the rescue.

Creating standardized parameter store paths for values based on the app or service we were deploying and consuming those cross stack using the native cloudformation parameter store parameter type gave us a lot of flexibility, allowed us to update resources as we needed and run the stack update on the consuming stacks to pick up new values

Cdk implementation

This is something I've started doing, and think it would make a great addition to aws-cdk. In aws-cdk it would probably be an addition to cdk.Stack to add methods for storing and retrieving values in parameter store, and probably look something like:

  /**
   * Store a resource value in parameter store, so it can be consumed by other stacks in the app
   * @param resource: cdk.Construct that owns the parameter you want to store
   * @param value: string the value to store in parameter store
   */
  store(resource: cdk.Construct, value: string): void {
    new ssm.StringParameter(this, `${resource.node.id}-parameter`, {
      parameterName: `/cdk/${this.stackName}/${resource.node.id}`,
      stringValue: value,
    });
  /**
   * Get a string value for a parameter stored by the persistent stack
   * @param sourceStack: the name of the stack that created the resource
   * @param resourceId: the `id` of the resource who's value you'd like to retrieve
   * @returns string the string value from the parameter store entry
   */
  getResource(sourceStack: string, resourceId: string): string {
    return ssm.StringParameter.valueForStringParameter(this, `/cdk/${SourceStack}/${resourceId}`);
  }

which could then be stored and retrieved like:

// in stack 1
    const table = new dynamodb.Table(this, 'myDynamoDBTable', {
      partitionKey: {
        name: 'someKey',
        type: dynamodb.AttributeType.STRING,
      },
    });
    this.store(table, table.tableArn);

// in stack 2
    const table = dynamodb.Table.fromTableArn(this, 'table', this.getResource('stack1', 'myDynamoDBTable'));

the consuming stack ends up with a cloudformation parameter which will resolve on stack update:

  SsmParameterValueappTemplateFargateServicetableC96584B6F00A464EAD1953AFF4B05118Parameter:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk/TemplateFargateService/myDynamoDBTable

[corrjo]

@skinny85 Would love to hear your thoughts.

[corrjo]

@NetaNir @eladb you guys might also be interested.
Also, is this the best place to start discussions like this?

[skinny85]

Hey @corrjo ,

thanks for opening the discussion. Also adding @rix0rrr.

This is something that we are aware of, and are planning to address, most likely using SSM Parameters, like you suggested (although I would guess our implementation would do all of that implicitly, without explicit store and getResource calls).

As for the format, we usually use issues for this kind of discuss (example issue about this: #9672), but I'm willing to give GitHub Discussions a shot 🙂.

Thanks,
Adam

[corrjo]

Thanks for your response @skinny85,
I agree, the explicit methods at too involved. After reading a bit more I imagine this is the behavior we'd want to change:

aws-cdk/packages/@aws-cdk/core/lib/private/refs.ts

Lines 90 to 103 in e09250b

	// ----------------------------------------------------------------------
	// export/import
	// ----------------------------------------------------------------------
	// export the value through a cloudformation "export name" and use an
	// Fn::ImportValue in the consumption site.
	// add a dependency between the producer and the consumer. dependency logic
	// will take care of applying the dependency at the right level (e.g. the
	// top-level stacks).
	consumer.addDependency(producer,
	`${consumer.node.path} -> ${reference.target.node.path}.${reference.displayName}`);
	return createImportValue(reference);

Changing the default behavior would be a breaking change for anyone using cross stack refs though. Probably something to put behind a feature flag?

Unless you're wanting to take this on in house, I'm happy to take a stab at implementing this if we agree on an approach.
John

[skinny85]

Changing the default behavior would be a breaking change for anyone using cross stack refs though. Probably something to put behind a feature flag?

Yep, absolutely.

Unless you're wanting to take this on in house, I'm happy to take a stab at implementing this if we agree on an approach.

We're absolutely not hell bent on doing these changes ourselves, it's just that these require going pretty deep into the CDK internals, and that's not usually something we can expect from the community. But I see you've already found where it should be changed, so you're half-way there! 🙂

Feel free to try and go all the way! If you stumble into any problems, let us know, and we'll try and help however we can 🙂.

Thanks so much for your engagement on this @corrjo!

[skinny85]

Here's a good example of a change using a feature flag: #11918, feel free to use it as a jumping-off point.

[corrjo]

Opened aws/aws-cdk-rfcs#311 for continued discussion

[henrist]

A use-case I'd like to see solved is better handling of resources across stages, by using a well-known format/name/storage for accessing resources across this boundary.

As an example, consider splitting a system in multiple stages or even multiple CDK applications, where a "central" part is responsible for a hosted zone. Another part might want to add a resource to this hosted zone, but it needs to be aware of its ID and zone name. One way to solve this is to use HostedZone.fromLookup(), but this requires the knowledge of an environment-specific value only known after a deployment, hard-coded into the repository, instead of using a well-known contract between the producer stack and the consumer stack.

Building on the idea of fromLookup(), would it be possible to have a fromParameterStore() method that would encapsulate the required parameters needed to use fromHostedZoneAttributes() or similar? To make the construct easily available in another stage, a CDK construct could implement exportToParameterStore() or similar to abstract away these details.

It seems @rix0rrr have already discussed this type of solution: #1095 (comment)

I think this could also be used to implement cross-region access to a construct or resource, by allow the storeAsParameter() to accept regions where the construct should be resolvable. I've implemented something similar to this: https://github.com/henrist/cdk-cross-region-params

There would still be limitations, such as the Vpc construct, that needs to know deploy-time specific information (such as the number/types of subnets in use).

Some related issues:

• Cross-region/account references #49
• Exporting/importing across CDK apps #1095

Some concerns:

• This will add to the number of CloudFormation parameters (likely not a big issue since the hard limit was recently changed from 60 to 200)
• Loosely coupling resources will likely lead to users doing more mistakes (such as deleting a resource being in use somewhere else)
• If a reference can be modified (which is prohibited when using CloudFormation Exports):
  • Downstream stacks would need to be redeployed - will likely cause (temporary) drift in many cases
  • Using the CDK CLI, I think the cdk deploy command of a downstream stack will skip CloudFormation deployment if the template is not changed (it will violate the deterministic idea of a template) - so it might not even pick up the change