Connection refused

[thesimpledata]

Getting the below exception. Using IBM cloud Environment Name:
IBM RedHat Openshift Kubernetes Service (VPC Gen2 with ODF) . Appears to be it does not have etcd cluster.

shakilkhan@Shakils-MacBook-Pro cerberus % source monitor/bin/activate
(monitor) shakilkhan@Shakils-MacBook-Pro cerberus % python3 start_cerberus.py --config /Users/shakilkhan/TECHZONE/TEC/Cerberus/config/config.yaml >> log.log
2022-06-12 11:33:59,197 [INFO] Starting ceberus
2022-06-12 11:33:59,206 [INFO] Initializing client to talk to the Kubernetes cluster
2022-06-12 11:33:59,762 [INFO] Fetching cluster info
2022-06-12 11:33:59,877 [INFO] Cluster version is 4.10.15
2022-06-12 11:33:59,877 [INFO] Server URL: https://c111-e.us-east.containers.cloud.ibm.com:32653
2022-06-12 11:33:59,877 [INFO] Publishing cerberus status at http://0.0.0.0:8080
2022-06-12 11:34:04,880 [INFO] Starting http server at http://0.0.0.0:8080

2022-06-12 11:34:09,399 [INFO] Daemon mode enabled, cerberus will monitor forever
2022-06-12 11:34:09,400 [INFO] Ignoring the iterations set

WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11a3f1fa0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-ingress/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116298dc0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-monitoring/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x122b6d790>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-machine-api/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116369a60>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-apiserver/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11a4040d0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-ingress/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116369b50>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-apiserver/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x122b6d880>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-machine-api/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116298eb0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-monitoring/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11a404250>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-ingress/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116369cd0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-apiserver/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x122b6da00>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-machine-api/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1162b1070>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-monitoring/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1163894c0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-etcd/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11a404490>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-scheduler/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1162b12b0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-apiserver/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x122b8d250>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-controller-manager/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11a404b20>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-scheduler/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116389580>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-etcd/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1162b1940>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-apiserver/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x122b8d340>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-controller-manager/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11a404ca0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-scheduler/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x122b8d4c0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-controller-manager/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1162b1ac0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-apiserver/pods?pretty=True&limit=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x116389700>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-etcd/pods?pretty=True&limit=
2022-06-12 11:34:09,462 [INFO] Encountered issues in cluster. Hence, setting the go/no-go signal to false
2022-06-12 11:34:09,479 [INFO] Exception: None: Max retries exceeded with url: /api/v1/namespaces/openshift-apiserver/pods?pretty=True&limit= (Caused by None)

2022-06-12 11:34:09,480 [INFO] SHAKIL
2022-06-12 11:34:09,480 [ERROR] None: Max retries exceeded with url: /api/v1/namespaces/openshift-apiserver/pods?pretty=True&limit= (Caused by None)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1253, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1299, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1248, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1008, in _send_output
self.send(msg)
File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 948, in send
self.connect()
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/Users/shakilkhan/TECHZONE/TEC/cerberus/monitor/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x116369e20>: Failed to establish a new connection: [Errno 61] Connection refused

[chaitanyaenr]

Hi @thesimpledata, thanks for reporting the issue. Looking at the logs, there seems to be issues with connecting with the cluster APIs: Failed to establish a new connection: [Errno 61] Connection refused')': /api/v1/namespaces/openshift-kube-apiserver/pods?pretty=True&limit=.

Also we might want to make sure the openshift user being used has appropriate permissions.

Thoughts?

[thesimpledata]

The user I am using is the cluster-admin . do i need any explicit rolebinding to run the cluster API?

/

Name

/
Name
Role ref
Subject kind
Subject name
Namespace
ClusterRoleBindingCRB
cluster-admin-0
ClusterRoleCR
cluster-admin
User IAM#[email protected] All namespaces
ClusterRoleBindingCRB
ibm-admin
ClusterRoleCR
cluster-admin
User IAM#[email protected] All namespaces
RoleBindingRB
user-settings-195869fc-dcda-4993-8a3d-4748513a421d-rolebinding
RoleR
user-settings-195869fc-dcda-4993-8a3d-4748513a421d-role
User IAM#[email protected]

[thesimpledata]

My kubeconfig looks like this

apiVersion: v1
clusters:

• cluster:
  server: https://c114-e.us-south.containers.cloud.ibm.com:31215
  name: c114-e-us-south-containers-cloud-ibm-com:31215
  contexts:
• context:
  cluster: c114-e-us-south-containers-cloud-ibm-com:31215
  namespace: default
  user: IAM#[email protected]/c114-e-us-south-containers-cloud-ibm-com:31215
  name: default/c114-e-us-south-containers-cloud-ibm-com:31215/IAM#[email protected]
  current-context: default/c114-e-us-south-containers-cloud-ibm-com:31215/IAM#[email protected]
  kind: Config
  preferences: {}
  users:
• name: IAM#[email protected]/c114-e-us-south-containers-cloud-ibm-com:31215
  user:
  token: sha256~9TcSTiu9mc3TpMaKNB9IJBZFqK-EELgLmKL4e8_rytA

Now in order for cluster API to work I need to construct the url as below

curl -H "Authorization: Bearer sha256~9TcSTiu9mc3TpMaKNB9IJBZFqK-EELgLmKL4e8_rytA" "https://c114-e.us-south.containers.cloud.ibm.com:31215/api/v1/nodes?limit=1"

I did not dig that far but I believe cerberus is not composing that url properly

[thesimpledata]

i am using IBM CLOUD

[ghost]

@thesimpledata this is not a permission error, this is a connection issue. Wherever you are running krkn, you don't seem to have network access to the OpenShift API.

[thesimpledata]

Not true. On the same cluster Kraken is working fine. Again I can manually run the API.

(monitor) shakilkhan@Shakils-MacBook-Pro cerberus % curl -H "Authorization: Bearer sha256~9TcSTiu9mc3TpMaKNB9IJBZFqK-EELgLmKL4e8_rytA" "https://c114-e.us-south.containers.cloud.ibm.com:31215/api/v1/nodes?limit=1"
{
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"resourceVersion": "3336359",
"continue": "eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6MzMzNjM1OSwic3RhcnQiOiIxMC4zOC4yMDAuMTQ3XHUwMDAwIn0",
"remainingItemCount": 6
},
"items": [
{
"metadata": {
"name": "10.38.200.147",
"uid": "b83436da-066a-4a5a-bd63-826f62116919",
"resourceVersion": "3335771",
"creationTimestamp": "2022-06-10T22:07:40Z",
"labels": {
"arch": "amd64",
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "c3c.16x32.encrypted",
"beta.kubernetes.io/os": "linux",
"failure-domain.beta.kubernetes.io/region": "us-south",
"failure-domain.beta.kubernetes.io/zone": "dal10",
"ibm-cloud.kubernetes.io/encrypted-docker-data": "true",
"ibm-cloud.kubernetes.io/external-ip": "169.63.200.123",
"ibm-cloud.kubernetes.io/iaas-provider": "softlayer",
"ibm-cloud.kubernetes.io/internal-ip": "10.38.200.147",
"ibm-cloud.kubernetes.io/machine-type": "c3c.16x32.encrypted",
"ibm-cloud.kubernetes.io/os": "REDHAT_7_64",
"ibm-cloud.kubernetes.io/region": "us-south",
"ibm-cloud.kubernetes.io/sgx-enabled": "false",
"ibm-cloud.kubernetes.io/worker-id": "kube-cahrog8d0id6oitvgmcg-itzroks6660-default-000003b4",
"ibm-cloud.kubernetes.io/worker-pool-id": "cahrog8d0id6oitvgmcg-1f3be9f",
"ibm-cloud.kubernetes.io/worker-pool-name": "default",
"ibm-cloud.kubernetes.io/worker-version": "4.8.42_1559_openshift",
"ibm-cloud.kubernetes.io/zone": "dal10",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "10.38.200.147",
"kubernetes.io/os": "linux",
"node-role.kubernetes.io/master": "",
"node-role.kubernetes.io/worker": "",
"node.kubernetes.io/instance-type": "c3c.16x32.encrypted",
"node.openshift.io/os_id": "rhel",
"privateVLAN": "2972490",
"publicVLAN": "2972492",
"topology.kubernetes.io/region": "us-south",
"topology.kubernetes.io/zone": "dal10"
},
"annotations": {
"projectcalico.org/IPv4Address": "10.38.200.147/26",
"projectcalico.org/IPv4IPIPTunnelAddr": "172.30.22.0"
},
"managedFields": [
{
"manager": "kube-controller-manager",

[thesimpledata]

Also I found issues with the global variable use in the client.py. It does not work