Authentication issue with ICP UI

[barsaboowo]

Hi,

After installing eventstreams-dev on our ICP installation we are unable to perform any tasks via the UI. For instance attempting to create a topic will result in a dialog box containing the following error:

Topic creation failed
500: An unexpected condition has occurred. org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed, invalid credentials
9/20/2019, 12:19:54 PM

[dalelane]

@barsaboowo Sorry to hear this!

Did you provide a custom certificate during your install (for either ICP and/or IBM Event Streams) ?

[barsaboowo]

Hi,

No, the default settings for certificates were used (self signed).

Thanks.

[barsaboowo]

@dalelane fyi this was installed from the helm charts catalog configuration UI using ibm-eventstreams-dev V 1.3.0.

[harveyelsom]

Hi @barsaboowo
Would you be able to run the scripts here https://github.com/IBM/event-streams/tree/master/support and provide us with the output? This should provide us with sufficient information to more thoroughly investigate your problem.
Thanks

[barsaboowo]

./installation-diagnostic-script.sh
Namespace : kafka-event-streams
Chart releasename : ibm-eventstreams-dev-1.3.0

---

Printing release candidate tag
Error from server (NotFound): deployments.extensions "ibm-eventstreams-dev-1.3.0-ibm-es-access-controller-deploy" not found

---

Printing current state
NAME READY STATUS RESTARTS AGE
spl-events-856c-ibm-es-kafka-sts-0 5/5 Running 10 5d23h
spl-events-856c-ibm-es-kafka-sts-1 5/5 Running 8 5d23h
spl-events-856c-ibm-es-kafka-sts-2 5/5 Running 8 5d23h
spl-events-856c-ibm-es-zook-c4c0-0 2/2 Running 2 5d23h
spl-events-856c-ibm-es-zook-c4c0-1 2/2 Running 2 5d23h
spl-events-856c-ibm-es-zook-c4c0-2 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf85fnmp 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf8smcpr 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-collector-deploy-554648fbd5-c2rrq 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-proxy-deploy-66867d4cb7-qp6q6 1/1 Running 5 5d23h
spl-eventstreams-poc4-ibm-es-rest-deploy-7bff5d695b-6h29k 3/3 Running 3 5d23h
spl-eventstreams-poc4-ibm-es-rest-producer-deploy-5cdbd44d62kqx 1/1 Running 1 5d23h
spl-eventstreams-poc4-ibm-es-rest-proxy-deploy-57bcf85dfc-9hbxg 1/1 Running 5 5d23h
spl-eventstreams-poc4-ibm-es-schemaregistry-sts-0 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-ui-deploy-c8b85f4c8-458s9 2/2 Running 2 5d23h

Starting release diagnostics

./installation-diagnostic-script.sh
Namespace : kafka-event-streams
Chart releasename : spl-eventstreams-poc4

---

Printing release candidate tag
ibm-eventstreams-dev-2019.2.1

Printing current state
NAME READY STATUS RESTARTS AGE
spl-events-856c-ibm-es-kafka-sts-0 5/5 Running 10 5d23h
spl-events-856c-ibm-es-kafka-sts-1 5/5 Running 8 5d23h
spl-events-856c-ibm-es-kafka-sts-2 5/5 Running 8 5d23h
spl-events-856c-ibm-es-zook-c4c0-0 2/2 Running 2 5d23h
spl-events-856c-ibm-es-zook-c4c0-1 2/2 Running 2 5d23h
spl-events-856c-ibm-es-zook-c4c0-2 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf85fnmp 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf8smcpr 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-collector-deploy-554648fbd5-c2rrq 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-proxy-deploy-66867d4cb7-qp6q6 1/1 Running 5 5d23h
spl-eventstreams-poc4-ibm-es-rest-deploy-7bff5d695b-6h29k 3/3 Running 3 5d23h
spl-eventstreams-poc4-ibm-es-rest-producer-deploy-5cdbd44d62kqx 1/1 Running 1 5d23h
spl-eventstreams-poc4-ibm-es-rest-proxy-deploy-57bcf85dfc-9hbxg 1/1 Running 5 5d23h
spl-eventstreams-poc4-ibm-es-schemaregistry-sts-0 2/2 Running 2 5d23h
spl-eventstreams-poc4-ibm-es-ui-deploy-c8b85f4c8-458s9 2/2 Running 2 5d23h

Starting release diagnostics

..Checking kafka-sts pods
....kafka-sts pods found
..Checking zookeeper-sts pods
....zookeeper-sts pods found

..Checking registered OAuth Endpoints
....OAuth Secret correctly generated
..Checking the Registered OAuth Endpoints
Trusted URI prefixes: https://xxx:yyy/ https://xxx:yyy/ https://xxx:yyy/
Redirect URIs: https://xxx:yyy/oauth/callback https://xxx:yyy/oauth/callback https://xxx:yyy/oauth/callback
Post Logout Redirect URIs: https://xxx:yyy/console/logout https://xxx:yyy/console/logout https://xxx:yyy/console/logout
....OAuthEndpoint check complete

..Checking User's current Role
....User Invalid Request due to missing/invalid token running with "User do not have access to CRN" Role

..Checking the ibm-es-iam-secret API Key
....API Key found

..Checking the ibm-es-iam-secret Service ID
....Service ID found
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 502 100 502 0 0 80 0 0:00:06 0:00:06 --:--:-- 121

..Checking the registered security role mappings
Role mapping actions defined:
action: eventstreams.cluster.manage
action: eventstreams.cluster.read
action: eventstreams.group.manage
action: eventstreams.group.read
action: eventstreams.schema.manage
action: eventstreams.schema.read
action: eventstreams.schema.write
action: eventstreams.topic.manage
action: eventstreams.topic.read
action: eventstreams.topic.write
action: eventstreams.txn.update
....role mappings check complete.

..Checking AccessController logs for errors
!!-- 5290 Errors found in pod/spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf85fnmp log, see tmpLogs/pod/spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf85fnmp/access-controller.log
!!-- 5244 Errors found in pod/spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf8smcpr log, see tmpLogs/pod/spl-eventstreams-poc4-ibm-es-access-controller-deploy-6bf8smcpr/access-controller.log

..Checking for Pending pods
....No Pending pods found

..Checking for Failed pods
....No Failed pods found

..Checking for CrashLoopBackOff pods
....No CrashLoopBackOff pods found

=======================================================
Test Summary

Kafka/Zookeeper Pods passed
Oauth Endpoints passed
IAM Secret Api Key passed
IAM Secret Service ID passed
Role Mappings passed
Access Controller logs failed **
Pending Pods passed
Failed Pods passed
CrashLoopBackOff Pods passed

Problems found during release diagnostics check. Please review output to identify potential problems.
If unable to identify or fix problems, please contact support.

[harveyelsom]

Could you cloudctl login to the namespace of the installation and run the get-logs.sh from the link provided previously, that script will output a zip with the logs of the pods. This output will give us the basis of analysing your authentication Issue. The installation diagnostics script output you have provided gives an overview of potential basic problems, but for a more in depth problem such as an authentication issue (assuming that you expect to have the admin privileges of creating a topic) we will need further logging.
Thanks