Istio Proxy Fails to Access Generate Token

[azunna1]

Hi all, i've been experiencing a weird issue where the istio proxy fails to generate an access token, resulting in many pods not starting up.

Here's the log:

2023-08-26T06:06:10.777869Z	info	token	Prepared federated token request for aud "identitynamespace:bank-staging-pci-01.svc.id.goog:https://gkehub.googleapis.com/projects/50523767346/locations/global/memberships/bank-staging-compute"
2023-08-26T06:06:10.789266Z	error	token	federated token response does not have access token{"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}
2023-08-26T06:06:10.789736Z	warning	envoy config external/envoy/source/common/config/grpc_stream.h:163	StreamAggregatedResources gRPC config stream to xds-grpc closed: 16, transport: per-RPC creds failed due to error: token manager failed to generate access token: federated token response does not have access token. {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}	thread=17
2023-08-26T06:06:15.061022Z	info	token	Prepared federated token request for aud "identitynamespace:bank-staging-pci-01.svc.id.goog:https://gkehub.googleapis.com/projects/50523767346/locations/global/memberships/bank-staging-compute"
2023-08-26T06:06:15.069741Z	error	token	federated token response does not have access token{"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}
2023-08-26T06:06:15.070265Z	warning	envoy config external/envoy/source/common/config/grpc_stream.h:163	StreamAggregatedResources gRPC config stream to xds-grpc closed: 16, transport: per-RPC creds failed due to error: token manager failed to generate access token: federated token response does not have access token. {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}	thread=17
2023-08-26T06:06:17.616995Z	info	token	Prepared federated token request for aud "identitynamespace:bank-staging-pci-01.svc.id.goog:https://gkehub.googleapis.com/projects/50523767346/locations/global/memberships/bank-staging-compute"
2023-08-26T06:06:17.627166Z	error	token	federated token response does not have access token{"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}
2023-08-26T06:06:17.627657Z	warning	envoy config external/envoy/source/common/config/grpc_stream.h:163	StreamAggregatedResources gRPC config stream to xds-grpc closed: 16, transport: per-RPC creds failed due to error: token manager failed to generate access token: federated token response does not have access token. {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}	thread=17
2023-08-26T06:06:22.411722Z	error	googleca	Failed to create certificate: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: token exchange failed: exchange failed all retries, last error: token exchange request failed: status code 400 body {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}, (aud: identitynamespace:bank-staging-pci-01.svc.id.goog:https://gkehub.googleapis.com/projects/50523767346/locations/global/memberships/bank-staging-compute, STS endpoint: https://sts.googleapis.com/v1/token)
2023-08-26T06:06:22.411780Z	warn	sds	failed to warm certificate: failed to generate workload certificate: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: token exchange failed: exchange failed all retries, last error: token exchange request failed: status code 400 body {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}, (aud: identitynamespace:bank-staging-pci-01.svc.id.goog:https://gkehub.googleapis.com/projects/50523767346/locations/global/memberships/bank-staging-compute, STS endpoint: https://sts.googleapis.com/v1/token)
2023-08-26T06:06:26.880248Z	error	Request to probe app failed: Get "http://10.40.3.18:10254/healthz": dial tcp 127.0.0.6:0->10.40.3.18:10254: connect: connection refused, original URL path = /app-health/controller/readyz
a

I've tried the following but nothing works:

• Unregistering the cluster from the fleet and registering it again
• Recreated the cluster
• Disabling the apis in the affected project and re-enabling it.
• Deleted the Fleet Project and Created a new one

I'm only left with the option of recreating the affected project.

I can't seem to get it to work which is really weird.
Here are the anthos installation details:

• Release Channel - Regular
• Version: 1.16.7-asm.4

PS: I'm using Cloud DNS as the DNS provider for the affected cluster and i had to update the cluster domain, it was after doing this i started experiencing this issue.

[zerobfd]

I don't think this is something that can be debugged properly via just a GitHub issue. Could you open a support ticket with Cloud so that the right set of people can all collaborate? You can just copy/paste the same information there.

[vedantthapa]

Hey there, I'm facing a similar issue.

@azunna1 by any chance did you resolve this? If so, could you please share the solution.