Monitoring feature is not working.

[artemnikitin2023]

Hello.
I've successfully deployed Bridge v4.16 on vanilla k8s with oidc auth. All works fine except the monitoring feature, there is no Observe\Monitoring UI Tab at all and metrics at Compute - Nodes is not available (got 404 not found error).
The monitoring plugin is deployed as a sidecar container in console pod, nginx listens localhost within a pod and the plugin is mapped via BRIDGE_PLUGINS="monitoring-plugin=http://localhost" environment variable. Also, i've set BRIDGE_PROMETHEUS_PUBLIC_URL, BRIDGE_ALERTMANAGER_PUBLIC_URL, BRIDGE_GRAFANA_PUBLIC_URL to the in cluster services (.svc.cluster.local), but looks like it's do nothing. There is no any error messages in logs so i don't understand why it's not working.

Is there anyone who have the similar setup and made it work? Please help.
Regards.

[Vovanys]

@artemnikitin2023
мне помогло скачивание офф образа
registry.redhat.io/openshift4/ose-monitoring-plugin-rhel9:v4.16.0-202408070337.p0.gaa339e4.assembly.stream.el9
после этого все заработало

[cloudcafetech]

@artemnikitin2023
Are you able to fix monitoring issue? Can you please share deployment ...

[Vovanys]

@cloudcafetech monitoring-plugin.zip
Copy this static to webserver (nginx, apache) and set url in env deployment console BRIDGE_PLUGINS="monitoring-plugin=http://static-url/folder"

[cloudcafetech]

@Vovanys

Will try & let you know ... Tnx :)

[cloudcafetech]

@Vovanys

No luck :(

• Getting error console pod log

E1012 05:34:09.401603       1 handlers.go:173] GET request for "monitoring-plugin" plugin failed with 304 status code
E1012 05:34:09.401625       1 utils.go:41] Failed sending HTTP response body: http: request method or response status code does not allow body
E1012 05:34:10.649623       1 handlers.go:173] GET request for "monitoring-plugin" plugin failed with 304 status code
E1012 05:34:10.649651       1 utils.go:41] Failed sending HTTP response body: http: request method or response status code does not allow body
E1012 05:34:25.324392       1 handlers.go:173] GET request for "monitoring-plugin" plugin failed with 304 status code
E1012 05:34:25.324423       1 utils.go:41] Failed sending HTTP response body: http: request method or response status code does not allow body

• Plugins log

192.168.1.220 - - [12/Oct/2024:05:37:46 +0000] "GET /plugin-manifest.json HTTP/1.1" 304 -
192.168.1.220 - - [12/Oct/2024:05:38:46 +0000] "GET /plugin-manifest.json HTTP/1.1" 304 -
192.168.1.220 - - [12/Oct/2024:05:39:24 +0000] "GET /plugin-manifest.json HTTP/1.1" 304 -
192.168.1.220 - - [12/Oct/2024:05:39:25 +0000] "GET /plugin-manifest.json HTTP/1.1" 304 -
192.168.1.220 - - [12/Oct/2024:05:39:40 +0000] "GET /plugin-manifest.json HTTP/1.1" 304 -

If I use 4:16 version, Console POD not coming up (error Fatal glibc error: CPU does not support x86-64-v2) But

If I use below version, Console POD coming up d But there is no Observe\Monitoring UI Tab .

[artemnikitin2023]

I've added some env variables to console container, after this the Observe menu appears. Also you need monitoring plugin sidecar container. Here is a variables set i'm using:

      env:
      - name: BRIDGE_PLUGINS
        value: "monitoring-plugin=http://localhost:80"
      - name: BRIDGE_USER_AUTH
        value: "oidc"
      - name: BRIDGE_USER_AUTH_OIDC_CLIENT_ID
        value: "your oidc client id"
      - name: BRIDGE_USER_AUTH_OIDC_CLIENT_SECRET
        value: "your oidc client secret"
      - name: BRIDGE_USER_AUTH_OIDC_ISSUER_URL
        value: "oidc issuer url for example https://dex.example.com/"
      - name: BRIDGE_K8S_AUTH
        value: "oidc"
      - name: BRIDGE_USER_AUTH_OIDC_TOKEN_SCOPES
        value: "additional oidc token scopes for your environment"
      - name: BRIDGE_COOKIE_AUTHENTICATION_KEY_FILE
        value: "path to mounted generated secret"
      - name: BRIDGE_COOKIE_ENCRYPTION_KEY_FILE
        value: "path to mounted generated secret"
      - name: BRIDGE_USER_AUTH_OIDC_CA_FILE
        value: "path to mounted ca file in case of self signed certs"
      - name: BRIDGE_BASE_ADDRESS
        value: "https://console.example.com"
      - name: BRIDGE_V
        value: "log level"
      - name: BRIDGE_PROMETHEUS_PUBLIC_URL
        value: "https://prometheus.example.com"
      - name: BRIDGE_K8S_MODE_OFF_CLUSTER_THANOS
        value: "https://prometheus.example.com"
      - name: BRIDGE_THANOS_PUBLIC_URL
        value: "https://prometheus.example.com"
      - name: BRIDGE_ALERTMANAGER_PUBLIC_URL
        value: "https://alertmanager.example.com"
      - name: BRIDGE_GRAFANA_PUBLIC_URL
        value: "https://grafana.example.com"
      - name: BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT
        value: "https://kubernetes.default"
      - name: BRIDGE_K8S_MODE
        value: "off-cluster"
      - name: BRIDGE_K8S_MODE_OFF_CLUSTER_SKIP_VERIFY_TLS
        value: "true"

Also you will need to edit promQL queries to make all metrics work.

[cloudcafetech]

@artemnikitin2023
Great, By the way .. which OCP console version & what is the K8S version ?

[BlindB0]

Hi @artemnikitin2023!
Did you solve the monitoring problem in vanila k8s?
Can you share monitoring plugin sidecar container yaml code?

[gecube]

@artemnikitin2023 Hi ! I was able to get Observe tab on the left on open shift-console installed on top of vanilla cluster. Did you get any success?

[gecube]

@BlindB0

Can you share monitoring plugin sidecar container yaml code?

I changed the Dockerfile for 4.16 to this one:

FROM node:18.1.0-alpine AS builder

WORKDIR /usr/src/app

#RUN npm install --global yarn

ENV HUSKY=0

COPY package.json yarn.lock .
RUN yarn

COPY ./console-extensions.json ./tsconfig.json ./webpack.config.ts .
COPY ./locales ./locales
COPY ./src ./src
RUN yarn build

FROM amazonlinux:2023.6.20241121.0

RUN INSTALL_PKGS="nginx" && \
    dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
    rpm -V $INSTALL_PKGS && \
    yum -y clean all --enablerepo='*' && \
    chown -R 1001:0 /var/lib/nginx /var/log/nginx /run && \
    chmod -R ug+rwX /var/lib/nginx /var/log/nginx /run

#USER 1001

COPY --from=builder /usr/src/app/dist /usr/share/nginx/html

ENTRYPOINT ["nginx", "-g", "daemon off;"]

and then deployed it to the cluster. Everything was relatively trivial.