Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Doesn't recognize AWS_WEB_IDENTITY_TOKEN_FILE, botocore.exceptions.ProfileNotFound: The config profile (default) could not be found #122

Open
rfvermut opened this issue Jul 2, 2021 · 15 comments
Open

Doesn't recognize AWS_WEB_IDENTITY_TOKEN_FILE, botocore.exceptions.ProfileNotFound: The config profile (default) could not be found #122

rfvermut opened this issue Jul 2, 2021 · 15 comments
Labels
bug

Comments

@rfvermut
Copy link

rfvermut commented Jul 2, 2021

In AWS EKS environment with IRSA. In version 0.22.

AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
    self._credentials = self._components.get_component(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
    return botocore.credentials.create_credential_resolver(
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

Works fine in 0.21
@okigan
Copy link
Owner

okigan commented Jul 3, 2021

Thanks for reporting - I’ll have a look

@okigan okigan added the bug label Jul 3, 2021
@iainelder
Copy link
Contributor

@rfvermut How are your other CLI environment variables set?

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

What sections do you have in your CLI config file and credentials file?

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

@rfvermut
Copy link
Author

rfvermut commented Jul 3, 2021 via email

@iainelder
Copy link
Contributor

I'm unfamiliar with AWS EKS. Does IRSA refer to the IAM roles for service accounts feature?

I don't think the behavior is affecterd by AWS_WEB_IDENTITY_TOKEN_FILE so I'll ignore it for now.

The server at example.com doesn't appear to respond to PUT requests so lets change that to a GET request.

awscurl -X GET https://example.com

So here's what I think is happening.

The behavior occurs only when botocore is available to awscurl.

Without knowing more about your environment, I will be using pipx to install awscurl in virtualenv and use the awslibs optional dependency to ensure that botocore is available.

pipx install awscurl[awslibs]==0.22.0

With the example command, awscurl via botocore loads the "default" profile (because that's the default value set by the CLI parser The default value is not None as I had previously thought!).

Since you have no configuration, botocore fails to find that profile and so fails.

But what happens in the previous version?

pipx uninstall awscurl
pipx install awscurl[awslibs]==0.21.0

With the example command, on my local machine, I get a different error, but it's for the same reason; there are no credentials to be found.

$ awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/home/norm/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

If I execute it on an EC2 instance with an attached IAM role, then awscurl via botocore loads the instance credentials from the instance metadata service.

@okigan I looks like this behavior was broken by my PR #116 for AWS SSO profiles.

I had believed that the default value for profile was None, but actually it's hard-coded to "default". That changes how botocore's default credential resolution works. It stops it falling back to the instance metadata servce.

I think there is a way to support all the desired credentials sources, but it will take time to find a correct solution.

For now I think the right thing to do is to undo my changes and not include them until we find a way to test all the use cases.

@iainelder
Copy link
Contributor

@rfvermut I'm not aware of a way to workaround this with version 0.22.0. For now I suggest you specify 0.21.0 when you install the command.

@okigan
Copy link
Owner

okigan commented Jul 3, 2021

I'm unfamiliar with AWS EKS. Does IRSA refer to the IAM roles for service accounts feature?

I don't think the behavior is affecterd by AWS_WEB_IDENTITY_TOKEN_FILE so I'll ignore it for now.

The server at example.com doesn't appear to respond to PUT requests so lets change that to a GET request.

awscurl -X GET https://example.com

So here's what I think is happening.

The behavior occurs only when botocore is available to awscurl.

Without knowing more about your environment, I will be using pipx to install awscurl in virtualenv and use the awslibs optional dependency to ensure that botocore is available.

pipx install awscurl[awslibs]==0.22.0

With the example command, awscurl via botocore loads the "default" profile (because that's the default value set by the CLI parser The default value is not None as I had previously thought!).

Since you have no configuration, botocore fails to find that profile and so fails.

But what happens in the previous version?

pipx uninstall awscurl
pipx install awscurl[awslibs]==0.21.0

With the example command, on my local machine, I get a different error, but it's for the same reason; there are no credentials to be found.

$ awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/home/norm/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

If I execute it on an EC2 instance with an attached IAM role, then awscurl via botocore loads the instance credentials from the instance metadata service.

@okigan I looks like this behavior was broken by my PR #116 for AWS SSO profiles.

I had believed that the default value for profile was None, but actually it's hard-coded to "default". That changes how botocore's default credential resolution works. It stops it falling back to the instance metadata servce.

I think there is a way to support all the desired credentials sources, but it will take time to find a correct solution.

For now I think the right thing to do is to undo my changes and not include them until we find a way to test all the use cases.

@iainelder i will be backing out the SSO feature/commit and releasing bug fix build - that should buy us time to figure out what’s going on here.

@iainelder iainelder mentioned this issue Jul 3, 2021
Open
@okigan
Copy link
Owner

okigan commented Jul 3, 2021

Change reverted: #124, new release will follow shortly

@okigan
Copy link
Owner

okigan commented Jul 3, 2021

@rfvermut I am looking for a way to setup a test environment for this, is the following script sufficient to repo? and in which environment does that need to run?

docker run  -it python:3-buster /bin/bash -c "pip3 install awscli awscurl && awscurl -X GET https://example.com"

@rfvermut
Copy link
Author

rfvermut commented Jul 4, 2021

First, let me thank you reverting that change so fast. I have a gazillion non-centralized Jenkins scripts that refer to non-pinned install of awscurl and it affected half of our organization. My mistake, always pin your versions.

Second, the "just enough" setup for tests would look like this:

Get inside a container

docker run -ti --rm python:3-buster /bin/bash

Prepare environment

export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/marketingadminrole"
export AWS_WEB_IDENTITY_TOKEN_FILE="/tmp/test"
echo "garbage" > $AWS_WEB_IDENTITY_TOKEN_FILE
pip3 install awscli awscurl==0.23 (or 22)

0.23/0.21 will fail trying to decode garbage while doing AssumeRoleWithWebIdentity which means it understands identity tokens

root@369514c4e8e8:/# awscurl example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 394, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 421, in access_key
    self._refresh()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 513, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 529, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 670, in fetch_credentials
    return self._get_cached_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 680, in _get_cached_credentials
    response = self._get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 890, in _get_credentials
    return client.assume_role_with_web_identity(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidIdentityTokenException: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: The ID Token provided is not a valid JWT. (You may see this error if you sent an Access Token)

0.22 will die as usual.

root@369514c4e8e8:/# awscurl example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
    self._credentials = self._components.get_component(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
    return botocore.credentials.create_credential_resolver(
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

@okigan
Copy link
Owner

okigan commented Jul 5, 2021

@rfvermut thanks for repro script!
@iainelder I have an idea, why profile is not working: in the command line parser it is the only one (among AWS_* related variables) that has a default value (of 'default'), so I think removing that and adding (back) the Session(profile=profile) could make it work. Here is related change: a501a43

@iainelder
Copy link
Contributor

Commit a501a43 looks to me like it could work.

(It's what I was alluding to in my ending comments in #123 (comment))

How would you test it?

@okigan
Copy link
Owner

okigan commented Jul 6, 2021

@iainelder i've tested it by checking out and running a501a43.
Getting the InvalidIdentityTokenException error as shown in #122 (comment) using following script:

export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/marketingadminrole"
export AWS_WEB_IDENTITY_TOKEN_FILE="/tmp/test"
echo "garbage" > $AWS_WEB_IDENTITY_TOKEN_FILE
python -m awscurl

i am still looking for a way to test load_aws_config, but do not see a way to mock the relevant component.

@iainelder
Copy link
Contributor

@okigan have you looked at moto? it has mocks for many AWS services including STS.

We might be able to use it to mock responses from the STS service that we are authenticating against.

https://github.com/spulec/moto

I have some ideas here, but no time to implement them for a while yet :-)

@NeckBeardPrince
Copy link

This is still an issue.

@okigan
Copy link
Owner

okigan commented Mar 14, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@okigan @iainelder @NeckBeardPrince @rfvermut