Update AWS Route53 documentation to better explain the authentication options #1555

wallrj · 2024-09-11T11:25:45Z

Preview: https://deploy-preview-1555--cert-manager.netlify.app/docs/configuration/acme/dns01/route53/

The aim of this PR is to help users understand that fundamentally cert-manager needs to get an access key somehow,
so that it can authenticate to Route53.

There are legacy mechanisms, where it can use an IAM User and a long-term access key from a mounted file, or from an environment variable in the cert-manager Pod, or by loading it from a Secret.

There are best-practice mechanisms, where it can use an IAM Role and get a temporary access key from STS, either by explicitly sending a Kubernetes ServiceAccount token to STS (AssumeRoleWithWebIdentity).
Or by fetching a temporary access token from a local server such as IMDS on an EC2 node or from a Pod Identity agent in a DaemonSet Pod on a Kubernetes Node.

There are ambient mechanisms, where credentials are global to the cert-manager controller.
And there are non-ambient mechanisms, where the credentials can be unique to each Issuer or ClusterIssuer.

I have also added documentation about how cert-manager chooses an AWS region and how that region is used.

Fixes: #56
Fixes: #753
Fixes: cert-manager/cert-manager#4738

netlify · 2024-09-11T11:27:59Z

✅ Deploy Preview for cert-manager ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`4d8ddd2`
🔍 Latest deploy log	https://app.netlify.com/sites/cert-manager/deploys/6707978b0916e20008ab20dc
😎 Deploy Preview	https://deploy-preview-1555--cert-manager.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Signed-off-by: Richard Wall <[email protected]>

… difference between ambient and non-ambient credentials Signed-off-by: Richard Wall <[email protected]>

viatcheslavmogilevsky · 2024-10-08T21:30:34Z

content/docs/configuration/acme/dns01/route53.md

+  where cert-manager loads credentials from files (`~/.aws/config` and `~/.aws/credentials`) which are mounted into the cert-manager controller Pod.
+
+The advantage of ambient credentials is that they are easier to set up, well
+documented, and AWS provides ways to automate the configuration.


From https://deploy-preview-1555--cert-manager.netlify.app/docs/configuration/acme/dns01/route53/#non-ambient-credentials I don't see any blockers to automate non-ambient credentials as well

I removed those words. What I meant was that AWS provides helpers which inject the necessary environment variables into the cert-manager Pod, rather than the application team having to configure everything in the issuer.
But that's already covered by "easier to set up".

Signed-off-by: Richard Wall <[email protected]>

wallrj · 2024-10-09T07:40:52Z

content/docs/tutorials/getting-started-aws-letsencrypt/README.md

-You will configure cert-manager to use the [Let's Encrypt DNS-01 challenge protocol](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) with AWS Route53 DNS,
-using IAM Roles for Service Accounts (IRSA) to authenticate to AWS.
+You will configure cert-manager to use the [Let's Encrypt DNS-01 challenge protocol](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) with AWS Route53 DNS.
+You will authenticate to Route53 using a [dedicated Kubernetes ServiceAccount token](../../configuration/acme/dns01/route53.md#referencing-your-own-serviceaccount-within-in-an-issuer-or-clusterissuer).


At the time I wrote the tutorial, I thought that IRSA referred to any auth mechanism that used a Kubernetes ServiceAccount token.
I now think that IRSA more of a marketing product name which comprises annotated Kubernetes ServiceAccount + mutating webhook (to inject a projected service account volume) + Kubernetes ServiceAccount token (signed JWT) loaded from /var/run/aws/token.

This tutorial does not use the webhook or projected service account token volume.

In future it might be worth updating the tutorial to use the much simpler "Pod Identity" mechanism,
because that will probably be more familiar to AWS EKS users.

Signed-off-by: Richard Wall <[email protected]>

wallrj · 2024-10-09T08:30:59Z

content/docs/configuration/acme/dns01/route53.md

@@ -138,251 +490,39 @@ And the following trust relationship (Add AWS `Service`s as needed):
 }
 ```

-## Creating an Issuer (or `ClusterIssuer`)
+## Region


In #56 @stephen-dexda and @TBBle showed that the region field should be optional. We made it optional in cert-manager/cert-manager#7287 and updated the API docs there.
Here I have added the same background information to the the website documentation.

https://deploy-preview-1555--cert-manager.netlify.app/docs/configuration/acme/dns01/route53/#region

@stephen-dexda and @TBBle: Please give feedback if you have time.

It's been a while since I looked at this stuff (so EKS Pod Identities is new to me (and very neat!)), so I'm not in a position to validate what's written here beyond the linked documentation.

With that qualification, the Region section looks fine to me.

wallrj · 2024-10-09T08:38:45Z

content/docs/configuration/acme/dns01/route53.md

+> 📖 Read about [actions supported by Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/APIReference/API_Operations_Amazon_Route_53.html),
+> in the [Amazon Route 53 API Reference](https://docs.aws.amazon.com/Route53/latest/APIReference/Welcome.html).
+>
+> 📖 Learn how [`eksctl` can automatically create the cert-manager IAM policy](https://eksctl.io/usage/iam-policies/#cert-manager-policy), if you use EKS.

 ## Credentials


In #753 @AlverezYari wrote:

Route53 - AWS IAM Account Setup is confusing

I've tried to summarize all the options in this revision of the documentation.

@AlverezYari Please give feedback if you have time:

https://deploy-preview-1555--cert-manager.netlify.app/docs/configuration/acme/dns01/route53/#credentials

Signed-off-by: Richard Wall <[email protected]>

SgtCoDFish

Spotted a couple of minor things, but this looks like a great improvement.

I've not run any code for this (don't have easy access to dev env), just a visual review.

SgtCoDFish · 2024-10-09T08:41:02Z

content/docs/configuration/acme/dns01/route53.md

+You have two options:
+1. (Legacy) Use an [IAM User and a long-term access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+2. (Best Practice) Use an [IAM Role with temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#bp-workloads-use-roles).


suggestion: With a big caveat that I've not really done much AWSing in the last few years, I probably wouldn't say that using a User is "legacy" because it's required for a specific use case - if you're using Route53 but you're running outside of AWS.

If all my compute is hosted on, say, DigitalOcean, I think an IAM User with long lived credentials would be a fine way to go if my DNS was in AWS.

Maybe if we re-order the best practice IAM role one to the top (i.e. number 1 in this list), and then change legacy to "for use when running code outside of AWS" or something similar. Or we could mention the "outside of AWS" use case below? What do you think?

I agree. I've removed "legacy" and explained that it is appropriate to use that mechanism when cert-manager is outside AWS.

content/docs/configuration/acme/dns01/route53.md

…to use that Signed-off-by: Richard Wall <[email protected]>

wallrj · 2024-10-09T09:21:00Z

content/docs/configuration/acme/dns01/route53.md

+
+cert-manager supports multiple ways to get the access key and these can be categorized as either "ambient" or "non-ambient".
+
+### Ambient Credentials


In cert-manager/cert-manager#4738 @twe-syde requested better "Documentation about ambient-credentials" and that:

The documentation should mentioned at least these two flags and why they are needed for IRSA.
--issuer-ambient-credentials=true|false
--clusterissuer-ambient-credentials=true|false

I've tried to address that here.

https://deploy-preview-1555--cert-manager.netlify.app/docs/configuration/acme/dns01/route53/#ambient-credentials

@twe-syde Please give feedback if you have time.

wallrj · 2024-10-09T10:23:59Z

content/docs/configuration/acme/dns01/route53.md

+3. **(optional) Update file system permissions**
+
+   You may also need to modify the cert-manager `Deployment` with the correct file system permissions, so the `ServiceAccount` token can be read.
+
+   ```yaml
+   spec:
+     template:
+       spec:
+         securityContext:
+           fsGroup: 1001
+   ```
+
+   The cert-manager Helm chart provides a variable for modifying cert-manager's `Deployment` like so:
+
+   ```yaml
+   securityContext:
+     fsGroup: 1001
+   ```


I have labelled this as optional, because I did not need this when I tested IRSA on an EKS cluster.
The reason may be found in cert-manager/cert-manager#2147 (comment) where @TBBle wrote:

Mentioned on the k8s 1.19 for EKS release notes, I believe the fsGroup setting should not be needed for k8s 1.19 onwards, due to the implementation of kubernetes/enhancements#1598.

I think it may be specifically required when using Fargate, based on @iainlane 's feedback in #697.
I will leave it to Fargate users to help us clarify this part of the documentation in future PRs.

Looking back at my own comment, it was based on a direct comment in the release notes but I don't recall testing it myself.

You're no longer required to provide a security context for non-root containers that need to access the web identity token file for use with IAM roles for service accounts. For more information, see IAM roles for service accounts andproposal for file permission handling in projected service account volume on GitHub.

Looking at the relevant KEP (subsequently subsumed into this KEP), the effect indeed should have been that if you don't have a runAsUser set or an fsGroup set, the token permissions were 0644, as of k8s 1.19. (A quick tour through the PR and the code in k8s repo HEAD shows this to still be the case, nothing jumped out as having changed since that PR.)

So I'm not sure what was up in #697, which was on k8s 1.21. Maybe fsGroup doesn't/didn't work properly on Fargate? I haven't found any direct references to this, except in the context of Fargate-hosted Pods with EFS mounts also requiring runAsUser and runAsGroup, to match the EFS configuration; I'm pretty sure that's an EFS-related issue though, and shouldn't have any bearing here. It might have been a bug in the Fargate implementation/packaging of kubelet?

Anyway, your testing says it's no longer needed, and that behaviour matches what passes for documentation in k8s, so I'm happy to see this demoted to "optional".

It might make sense to change the advice to use runAsUser: 1000 since the Docker container runs as USER 1000 by default. It would be relatively harmless to add that the Helm chart defaults too, since that's effectively just documenting something already defined in the container image. We already use runAsNonRoot, but sadly that does not interact with the projected volume mounting system as at that point, the actual in-pod user ID has not yet been extracted/verified. There's no harm in having both runAsNonRoot and runAsUser, and fsGroup has a (minimal in this case) runtime cost.

Added a note to encourage users to tell us once and for all whether this step is obsolete.

@SgtCoDFish I see that you also asked whether this step is necessary:

Improve docs for fargate networking to avoid confusion #698 (comment)

Along with a call for feedback from EKS Fargate users Signed-off-by: Richard Wall <[email protected]>

SgtCoDFish

/lgtm
/approve

I've not run any code or tested this, but the text looks like a clear improvement so it makes sense to merge. Thanks for all the work on this @wallrj and thanks everyone who commented and got involved 🚀

cert-manager-prow · 2024-10-10T10:08:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SgtCoDFish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [SgtCoDFish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wallrj force-pushed the aws-route53-authentication-summary branch from d8de949 to 251cd75 Compare September 11, 2024 12:07

cert-manager-prow bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 11, 2024

wallrj force-pushed the aws-route53-authentication-summary branch 6 times, most recently from 57d106f to 6668546 Compare September 12, 2024 08:20

This was referenced Sep 20, 2024

Allow the Route53 solver to get the AWS region from IMDS when running on EKS or EC2 cert-manager/cert-manager#7293

Draft

Missing RBAC for cert-manager default serviceaccount cert-manager/cert-manager#7212

Closed

wallrj force-pushed the aws-route53-authentication-summary branch 3 times, most recently from 10bf1cb to 2b85a62 Compare October 8, 2024 14:27

wallrj added 4 commits October 8, 2024 15:35

Fix the heading to say Policy instead of Role

ccfe3ad

Signed-off-by: Richard Wall <[email protected]>

Use an info icon to accentuate the note about Policy permissions

aa2a29b

Signed-off-by: Richard Wall <[email protected]>

Add some further links of interest in the Policy section

8d436a1

Signed-off-by: Richard Wall <[email protected]>

Summarize the ways cert-manager can get an access key and explain the…

c989f4c

… difference between ambient and non-ambient credentials Signed-off-by: Richard Wall <[email protected]>

wallrj force-pushed the aws-route53-authentication-summary branch from 2b85a62 to c989f4c Compare October 8, 2024 16:10

cert-manager-prow bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 8, 2024

wallrj marked this pull request as ready for review October 8, 2024 16:22

wallrj changed the title ~~WIP: Update AWS Route53 documentation to better explain the authentication options~~ Update AWS Route53 documentation to better explain the authentication options Oct 8, 2024

wallrj requested review from SgtCoDFish and removed request for maelvls October 8, 2024 19:32

viatcheslavmogilevsky reviewed Oct 8, 2024

View reviewed changes

Notes about the region field

b8b1a1c

Signed-off-by: Richard Wall <[email protected]>

wallrj mentioned this pull request Oct 9, 2024

Route53 Provider Assume Role Error - Missing Region cert-manager/cert-manager#7102

Closed

wallrj force-pushed the aws-route53-authentication-summary branch from cf812de to e519b51 Compare October 9, 2024 07:32

wallrj commented Oct 9, 2024

View reviewed changes

Cross links with the tutorial

ae9a9e0

Signed-off-by: Richard Wall <[email protected]>

wallrj force-pushed the aws-route53-authentication-summary branch from e519b51 to ae9a9e0 Compare October 9, 2024 07:42

wallrj commented Oct 9, 2024

View reviewed changes

wallrj added 2 commits October 9, 2024 09:44

Link to the IRSA introduction

71a554d

Signed-off-by: Richard Wall <[email protected]>

Fix invalid yaml

7c5947f

Signed-off-by: Richard Wall <[email protected]>

SgtCoDFish requested changes Oct 9, 2024

View reviewed changes

Don't describe IAM User as legacy and explain when it is appropriate …

2f19d2e

…to use that Signed-off-by: Richard Wall <[email protected]>

wallrj commented Oct 9, 2024

View reviewed changes

wallrj requested a review from SgtCoDFish October 9, 2024 10:25

wallrj mentioned this pull request Oct 9, 2024

route53.role can be omitted when using IRSA #1487

Closed

wallrj force-pushed the aws-route53-authentication-summary branch 2 times, most recently from 8b96699 to dbdddfb Compare October 10, 2024 08:54

wallrj mentioned this pull request Oct 10, 2024

[IRSA] Needs runAsUser: 1001 #697

Open

Add runAsUser: 1001 to the IRSA instructions

4d8ddd2

Along with a call for feedback from EKS Fargate users Signed-off-by: Richard Wall <[email protected]>

wallrj force-pushed the aws-route53-authentication-summary branch from dbdddfb to 4d8ddd2 Compare October 10, 2024 08:59

SgtCoDFish approved these changes Oct 10, 2024

View reviewed changes

cert-manager-prow bot added the lgtm Indicates that a PR is ready to be merged. label Oct 10, 2024

cert-manager-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 10, 2024

cert-manager-prow bot merged commit 41b93f6 into cert-manager:master Oct 10, 2024
7 checks passed

wallrj deleted the aws-route53-authentication-summary branch October 10, 2024 10:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update AWS Route53 documentation to better explain the authentication options #1555

Update AWS Route53 documentation to better explain the authentication options #1555

wallrj commented Sep 11, 2024 •

edited

Loading

netlify bot commented Sep 11, 2024 •

edited

Loading

viatcheslavmogilevsky Oct 8, 2024

wallrj Oct 9, 2024

wallrj Oct 9, 2024

wallrj Oct 9, 2024 •

edited

Loading

TBBle Oct 9, 2024

wallrj Oct 9, 2024

SgtCoDFish left a comment

SgtCoDFish Oct 9, 2024

wallrj Oct 9, 2024

wallrj Oct 9, 2024

wallrj Oct 9, 2024

TBBle Oct 9, 2024 •

edited

Loading

wallrj Oct 10, 2024 •

edited

Loading

SgtCoDFish left a comment

cert-manager-prow bot commented Oct 10, 2024


		cert-manager supports multiple ways to get the access key and these can be categorized as either "ambient" or "non-ambient".

		### Ambient Credentials

Update AWS Route53 documentation to better explain the authentication options #1555

Update AWS Route53 documentation to better explain the authentication options #1555

Conversation

wallrj commented Sep 11, 2024 • edited Loading

netlify bot commented Sep 11, 2024 • edited Loading

✅ Deploy Preview for cert-manager ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wallrj Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SgtCoDFish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TBBle Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

wallrj Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

SgtCoDFish left a comment

Choose a reason for hiding this comment

cert-manager-prow bot commented Oct 10, 2024

wallrj commented Sep 11, 2024 •

edited

Loading

netlify bot commented Sep 11, 2024 •

edited

Loading

wallrj Oct 9, 2024 •

edited

Loading

TBBle Oct 9, 2024 •

edited

Loading

wallrj Oct 10, 2024 •

edited

Loading