Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

http addon using Ingress, NGINX Ingress Controller and ExternalName Service #993

Closed
shreya0910 opened this issue Apr 26, 2024 · 5 comments
Closed

http addon using Ingress, NGINX Ingress Controller and ExternalName Service #993

shreya0910 opened this issue Apr 26, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@shreya0910
Copy link

shreya0910 commented Apr 26, 2024 •
edited
Loading

Report

I am trying to setup keda + httpadd on is in keda ns & application + ingress is in app ns. Created Ingress, NGINX Ingress Controller and ExternalName Service setup(in app ns)

Here is the external service manifest

apiVersion: v1
kind: Service
  metadata:
    name: keda-add-ons-http-interceptor-proxy
    namespace: app
spec:
  type: ExternalName
  externalName: keda-add-ons-http-interceptor-proxy.keda.svc.cluster.local

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    artifact.spinnaker.io/location: app
    artifact.spinnaker.io/name: health-check-svc
    artifact.spinnaker.io/type: kubernetes/ingress
    artifact.spinnaker.io/version: ''
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /path/
  labels:
  ## some labels
  name: health-check-svc
  namespace: app
spec:
  rules:
    - host: test.com
      http:
        paths:
          - backend:
              service:
                name: keda-add-ons-http-interceptor-proxy
                port:
                  number: 8080
            path: /dshc/
            pathType: ImplementationSpecific

Scaledobject

apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  name: health-check-svc
  namespace: app
spec:
  hosts:
    - test.com
  pathPrefixes:
    - /path
  replicas:
    max: 5
    min: 0
  scaleTargetRef:
    deployment: health-check-svc
    name: health-check-svc
    port: 8080
    service: health-check-svc
  targetPendingRequests: 1

but so far its not working as expected
keda-add-ons-http-interceptor-proxy is not reachable from nginx. Here is the log

[26/Apr/2024:22:31:42 +0000] "GET /path HTTP/1.1" 499 0 "-" "-" 120 15.993 [apps-keda-add-ons-http-interceptor-proxy-8080] []
[26/Apr/2024:22:31:42 +0000] "GET /path HTTP/1.1" 502 150 "-" "-" 120 2.025 [apps-keda-add-ons-http-interceptor-proxy-8080] []

Expected Behavior

Request should be able to reach interceptor-proxy

Actual Behavior

[26/Apr/2024:22:31:42 +0000] "GET /path HTTP/1.1" 499 0 "-" "-" 120 15.993 [apps-keda-add-ons-http-interceptor-proxy-8080] []
[26/Apr/2024:22:31:42 +0000] "GET /path HTTP/1.1" 502 150 "-" "-" 120 2.025 [apps-keda-add-ons-http-interceptor-proxy-8080] []

Steps to Reproduce the Problem

  1. Install keda & keda http add on in keda ns
  2. Deploy application, ingress & external svc in app ns
  3. Try to curl the host

Logs from KEDA HTTP operator

example

operator logs

│ 2024-04-26T22:48:40Z    INFO    setup    Running on Kubernetes 1.26+    {"version": "v1.26.14-eks-b9c9ed7"}                                                                                                                                                                                                             │
│ 2024-04-26T22:48:40Z    INFO    setup    WARNING: KEDA 2.13.0 hasn't been tested on Kubernetes v1.26.14-eks-b9c9ed7                                                                                                                                                                                                     │
│ 2024-04-26T22:48:40Z    INFO    setup    You can check recommended versions on https://keda.sh                                                                                                                                                                                                                          │
│ 2024-04-26T22:48:40Z    INFO    controller-runtime.metrics    Starting metrics server                                                                                                                                                                                                                                   │
│ 2024-04-26T22:48:40Z    INFO    starting server    {"kind": "health probe", "addr": ":8081"}                                                                                                                                                                                                                            │
│ I0426 22:48:40.968974       1 leaderelection.go:250] attempting to acquire leader lease keda/operator.keda.sh...                                                                                                                                                                                                        │
│ 2024-04-26T22:48:40Z    INFO    controller-runtime.metrics    Serving metrics server    {"bindAddress": ":8080", "secure": false}                                                                                                                                                                                       │
│ I0426 22:49:23.897424       1 leaderelection.go:260] successfully acquired lease keda/operator.keda.sh                                                                                                                                                                                                                  │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}                                                                                                                 │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}                                                                                                            │
│ 2024-04-26T22:49:23Z    INFO    Starting Controller    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}                                                                                                                                                                   │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}                                                                                      │
│ 2024-04-26T22:49:23Z    INFO    Starting Controller    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}                                                                                                                                                 │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}                                                                                                                          │
│ 2024-04-26T22:49:23Z    INFO    Starting Controller    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}                                                                                                                                                                         │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "source": "kind source: *v1alpha1.CloudEventSource"}                                                                                            │
│ 2024-04-26T22:49:23Z    INFO    Starting Controller    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}                                                                                                                                                  │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}                                                                 │
│ 2024-04-26T22:49:23Z    INFO    Starting Controller    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}                                                                                                                                   │
│ 2024-04-26T22:49:23Z    INFO    cert-rotation    starting cert rotator controller                                                                                                                                                                                                                                       │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}                                                                                                                                                                                             │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}                                                                                                                                                                             │
│ 2024-04-26T22:49:23Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}                                                                                                                                                                             │
│ 2024-04-26T22:49:23Z    INFO    Starting Controller    {"controller": "cert-rotator"}                                                                                                                                                                                                                                   │
│     /workspace/controllers/keda/scaledobject_controller.go:264                                                                                                                                                                                                                                                          │
│ github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile                                                                                                                                                                                                                                        │
│     /workspace/controllers/keda/scaledobject_controller.go:191                                                                                                                                                                                                                                                          │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile                                                                                                                                                                                                                                          │
│     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119                                                                                                                                                                                                                          │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                                                                                                                                                                                                                                   │
│     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316                                                                                                                                                                                                                          │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                                                                                                                                                                                                                                │
│     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266                                                                                                                                                                                                                          │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                                                                                                                                                                                                                                      │
│     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227                                                                                                                                                                                                                          │
│ 2024-04-26T22:49:24Z    ERROR    ****ScaledObject doesn't have correct scaleTargetRef specification    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-http-helm-chart-interceptor","namespace":"keda"}, "namespace": "keda", "name": "keda-ht │
│ github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile****

interceptor logs

{"level":"info","ts":1714163786.658197,"logger":"LoggingMiddleware","caller":"middleware/logging.go:73","msg":"10.10.227.111:41246 - - [26/Apr/2024:20:36:26 +0000] \"GET /livez HTTP/1.1\" 200 2 \"\" \"kube-probe/1.26+\""}                                                                                           │
│ {"level":"info","ts":1714163786.6583877,"logger":"LoggingMiddleware","caller":"middleware/logging.go:73","msg":"10.10.227.111:41244 - - [26/Apr/2024:20:36:26 +0000] \"GET /readyz HTTP/1.1\" 200 2 \"\" \"kube-probe/1.26+\""}                                                                                         │
│ W0426 20:36:28.114572       1 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:keda:keda-http-helm-chart-interceptor" cannot list resource "deployments" in API group "apps" at the cluster scope     │
│ E0426 20:36:28.114602       1 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:keda:keda-http-helm-chart-interceptor" cannot list resource "deployments" in API group │

HTTP Add-on Version

0.6.0

Kubernetes Version

< 1.27

Platform

Amazon Web Services

Anything else?

No response
@shreya0910 shreya0910 added the bug Something isn't working label Apr 26, 2024
@JorTurFer
Copy link
Member

Hello!
From your manifests, I think that the externalName service is called demo-externalname but the service called from nginx is keda-add-ons-http-interceptor-proxy.
I assume that your ingress is in the application namespace, so the route is

  • ingress -> external name service -> http-add-on service -> interceptor -> you app service -> your app
    Try changing the service in the ingress to the externalName service

@JorTurFer
Copy link
Member

I can see also an issue related with RBAC in the interceptor, I'd suggest upgrading to v0.7.0. For going live, I'd wait until v0.8.0, because we found an error on helm chart that blocks the helm upgrade, forcing to remove and reinstall (so, for evaluating the add-on can be enough v0.7.0, but for something stable you should wait for v0.8.0)

v0.8.0 will be released soon, and your problem isn't related with the version but with ingress config

@shreya0910
Copy link
Author

Hello! From your manifests, I think that the externalName service is called demo-externalname but the service called from nginx is keda-add-ons-http-interceptor-proxy. I assume that your ingress is in the application namespace, so the route is

  • ingress -> external name service -> http-add-on service -> interceptor -> you app service -> your app
    Try changing the service in the ingress to the externalName service

Hello!! Sorry that was a typo. I have updated it.
Yes the route is this - ingress -> external name service -> http-add-on service -> interceptor -> you app service -> your app.. Is this standard approach in such scenario when ingress is in application ns? If possible could you share some references. Thanks

@JorTurFer
Copy link
Member

Is this standard approach in such scenario when ingress is in application ns?

Yeah, it's a good option indeed

Yes the route is this - ingress -> external name service -> http-add-on service -> interceptor -> you app service -> your app

Your ingress is pointing directly to http-add-on service, which probably isn't correct as you should refer to the external name service

apiVersion: networking.k8s.io/v1
kind: Ingress
...
spec:
  rules:
    - host: test.com
      http:
        paths:
          - backend:
              service:
                name: keda-add-ons-http-interceptor-proxy
                ....

Is that your configuration at ingress level?

@shreya0910
Copy link
Author

Is this standard approach in such scenario when ingress is in application ns?

Yeah, it's a good option indeed

Yes the route is this - ingress -> external name service -> http-add-on service -> interceptor -> you app service -> your app

Your ingress is pointing directly to http-add-on service, which probably isn't correct as you should refer to the external name service

apiVersion: networking.k8s.io/v1
kind: Ingress
...
spec:
  rules:
    - host: test.com
      http:
        paths:
          - backend:
              service:
                name: keda-add-ons-http-interceptor-proxy
                ....

Is that your configuration at ingress level?

Thanks for your inputs. I am closing this issue now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA HTTP Add-On
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JorTurFer @shreya0910