Skip to content

Latest commit

 

History

History
98 lines (74 loc) · 4.29 KB

README.md

File metadata and controls

98 lines (74 loc) · 4.29 KB
Raw

external-metrics-multiplexer

External Metrics is the blessed Kubernetes way of retrieving metrics and exposing them to the Kubernetes API to provide data for autoscaling decisions.

However, by design, only one provider of external metrics can be used at any given time.

There is an open proposal to fix this (kubernetes-sigs/custom-metrics-apiserver#70), but in the meantime: please meet the external-metrics-multiplexer!

external-metrics-multiplexer allows running several External Metrics providers in Kubernetes at once, and reverse-proxies to providers based on metric prefixes.

Disclaimer

This is a bit of a hackish proof-of-concept, and while it works, it's not particularly secure (see Design) or well-tested. If you plan to use this in prod, I'd advise rewriting it to address the security implications, or use a more fully-fledged project like Keda.

Design

Essentially, this chart is just a souped-up nginx that:

  • Registers itself as the external metrics APIService
  • Matches metrics to a metric provider based on their prefix
  • Optionally, strips the prefix
  • Reverse-proxies the metric request to its metric provider

Regarding authentication/authorization: in order to accept HTTPS requests from the APIServer, we generate a self-signed certificates at start-up time and set nginx to use it.

⚠️ This means we use insecureSkipTLSVerify: true in the APIService, which is probably ok but obviously not recommended.

Then, to talk to the metrics providers, we create a ServiceAccount with permissions to read metrics (see templates/rbac.yaml) and use its ServiceAccount token to communicate with the backends. This is required (rather than passing the token of the requester) because some requesters use X.509 certificates which cannot be passed through (at least not without modifying the backend services to parse them out of a header).

⚠️ This means cluster-local workloads could see the metrics without RBAC

Configure & install

Configure the providers you want to use in values.yaml, install that Helm chart, and you're good to go!

Once installed, you'll be able to use several metrics providers in HPAs, like:

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-prometheus
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker-prometheus
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: "prometheus-a-metric-called-foo" # Will use "a-metric-called-foo" from the Prometheus provider
        selector:
          matchLabels:
            yeah: "labels-should-work-too"
      target:
        type: AverageValue
        value: 1

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-cloudwatch
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker-cloudwatch
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: "cloudwatch-bar" # Will use metric "bar" from the Cloudwatch provider
      target:
        type: AverageValue
        value: 1

Supported providers

external-metrics-multiplexer has been tested with the following providers:

Other compliant external-metrics providers should work as well.

Known issues

external-metrics-multiplexer currently doesn't work with Datadog.

As far as I understand, this is due to Datadog-specific optimisations. Datadog lists all the HPAs, then creates one huge Datadog query by concatenating all the various metrics, and sends this to the Datadog API in a single request.

This is all very clever and optimised, but unfortunately it'll break if any query is not datadog-compatible, which is pretty much always the case when using multiple providers.