Skip to content

Commit

Permalink
feat: OB-36454 update values file to new conventions and add and upda…
Browse files Browse the repository at this point in the history
…te tests

- Update values file to give more descriptive names and add comments for clarity
- Add new examples for prometheus labels
  • Loading branch information
arthur-observe committed Oct 1, 2024
1 parent 74bbf39 commit d643670
Show file tree
Hide file tree
Showing 8 changed files with 235 additions and 31 deletions.
2 changes: 1 addition & 1 deletion charts/agent/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: agent
description: Chart to install K8s collection stack based on Observe Agent
type: application
version: 0.18.0
version: 0.19.0
appVersion: "1.1.0"
dependencies:
- name: opentelemetry-collector
Expand Down
21 changes: 12 additions & 9 deletions charts/agent/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# agent

![Version: 0.18.0](https://img.shields.io/badge/Version-0.18.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.1.0](https://img.shields.io/badge/AppVersion-1.1.0-informational?style=flat-square)
![Version: 0.19.0](https://img.shields.io/badge/Version-0.19.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.1.0](https://img.shields.io/badge/AppVersion-1.1.0-informational?style=flat-square)

> [!CAUTION]
> This chart is under active development and is not meant to be installed yet.
Expand Down Expand Up @@ -89,6 +89,7 @@ Chart to install K8s collection stack based on Observe Agent
| cluster-events.podAnnotations.observe_monitor_port | string | `"8888"` | |
| cluster-events.podAnnotations.observe_monitor_purpose | string | `"observecollection"` | |
| cluster-events.podAnnotations.observe_monitor_scrape | string | `"true"` | |
| cluster-events.podAnnotations.observeinc_com_scrape | string | `"false"` | |
| cluster-events.ports.metrics.containerPort | int | `8888` | |
| cluster-events.ports.metrics.enabled | bool | `true` | |
| cluster-events.ports.metrics.protocol | string | `"TCP"` | |
Expand Down Expand Up @@ -152,6 +153,7 @@ Chart to install K8s collection stack based on Observe Agent
| cluster-metrics.podAnnotations.observe_monitor_port | string | `"8888"` | |
| cluster-metrics.podAnnotations.observe_monitor_purpose | string | `"observecollection"` | |
| cluster-metrics.podAnnotations.observe_monitor_scrape | string | `"true"` | |
| cluster-metrics.podAnnotations.observeinc_com_scrape | string | `"false"` | |
| cluster-metrics.ports.metrics.containerPort | int | `8888` | |
| cluster-metrics.ports.metrics.enabled | bool | `true` | |
| cluster-metrics.ports.metrics.protocol | string | `"TCP"` | |
Expand All @@ -166,14 +168,13 @@ Chart to install K8s collection stack based on Observe Agent
| cluster.events.enabled | bool | `true` | |
| cluster.events.pullInterval | string | `"20m"` | |
| cluster.metrics.enabled | bool | `true` | |
| cluster.metrics.pod.action | string | `"keep"` | |
| cluster.metrics.pod.enabled | bool | `false` | |
| cluster.metrics.pod.interval | string | `"10s"` | |
| cluster.metrics.pod.metric_drop_regex | string | `".*bucket"` | |
| cluster.metrics.pod.metric_keep_regex | string | `"(.*)"` | |
| cluster.metrics.pod.namespace_drop_regex | string | `"(.*istio.*|.*ingress.*|kube-system|observe)"` | |
| cluster.metrics.pod.namespace_keep_regex | string | `"(.*)"` | |
| cluster.metrics.pod.port_keep_regex | string | `".*metrics"` | |
| cluster.metrics.prometheusScrape.enabled | bool | `false` | |
| cluster.metrics.prometheusScrape.interval | string | `"10s"` | |
| cluster.metrics.prometheusScrape.metricDropRegex | string | `".*bucket"` | |
| cluster.metrics.prometheusScrape.metricKeepRegex | string | `"(.*)"` | |
| cluster.metrics.prometheusScrape.namespaceDropRegex | string | `"(.*istio.*|.*ingress.*|kube-system)"` | |
| cluster.metrics.prometheusScrape.namespaceKeepRegex | string | `"(.*)"` | |
| cluster.metrics.prometheusScrape.portKeepRegex | string | `".*metrics"` | |
| cluster.name | string | `"observe-agent-monitored-cluster"` | |
| cluster.namespaceOverride.value | string | `"observe"` | |
| cluster.uidOverride.value | string | `""` | |
Expand Down Expand Up @@ -229,6 +230,7 @@ Chart to install K8s collection stack based on Observe Agent
| monitor.podAnnotations.observe_monitor_port | string | `"8888"` | |
| monitor.podAnnotations.observe_monitor_purpose | string | `"observecollection"` | |
| monitor.podAnnotations.observe_monitor_scrape | string | `"false"` | |
| monitor.podAnnotations.observeinc_com_scrape | string | `"false"` | |
| monitor.ports.metrics.containerPort | int | `8888` | |
| monitor.ports.metrics.enabled | bool | `true` | |
| monitor.ports.metrics.protocol | string | `"TCP"` | |
Expand Down Expand Up @@ -315,6 +317,7 @@ Chart to install K8s collection stack based on Observe Agent
| node-logs-metrics.podAnnotations.observe_monitor_port | string | `"8888"` | |
| node-logs-metrics.podAnnotations.observe_monitor_purpose | string | `"observecollection"` | |
| node-logs-metrics.podAnnotations.observe_monitor_scrape | string | `"true"` | |
| node-logs-metrics.podAnnotations.observeinc_com_scrape | string | `"false"` | |
| node-logs-metrics.ports.metrics.containerPort | int | `8888` | |
| node-logs-metrics.ports.metrics.enabled | bool | `true` | |
| node-logs-metrics.ports.metrics.protocol | string | `"TCP"` | |
Expand Down
15 changes: 8 additions & 7 deletions charts/agent/templates/_cluster-metrics-config.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ receivers:
metrics:
k8s.node.condition:
enabled: true
{{- if .Values.cluster.metrics.pod.enabled }}

{{- if .Values.cluster.metrics.prometheusScrape.enabled }}
prometheus/pod_metrics:
config:
scrape_configs:
Expand All @@ -34,22 +35,22 @@ receivers:
- role: pod
relabel_configs:
# this is defaulted to keep but if set to drop then pod metrics will not be collected
- action: {{.Values.cluster.metrics.pod.action}}
- action: keep

# Drop anything matching the configured namespace.
- action: 'drop'
source_labels: ['__meta_kubernetes_namespace']
regex: {{.Values.cluster.metrics.pod.namespace_drop_regex}}
regex: {{.Values.cluster.metrics.pod.namespaceDropRegex}}

# Drop anything not matching the configured namespace.
- action: 'keep'
source_labels: ['__meta_kubernetes_namespace']
regex: {{.Values.cluster.metrics.pod.namespace_keep_regex}}
regex: {{.Values.cluster.metrics.pod.namespaceKeepRegex}}

# Drop endpoints without one of: a port name suffixed with the configured regex, or an explicit prometheus port annotation.
- action: 'keep'
source_labels: ['__meta_kubernetes_pod_container_port_name', '__meta_kubernetes_pod_annotation_prometheus_io_port']
regex: '({{.Values.cluster.metrics.pod.port_keep_regex}};|.*;\d+)'
regex: '({{.Values.cluster.metrics.pod.portKeepRegex}};|.*;\d+)'

# Drop pods with phase Succeeded or Failed.
- action: 'drop'
Expand Down Expand Up @@ -109,7 +110,7 @@ receivers:
- source_labels: [__address__, __meta_kubernetes_pod_annotation_observeinc_com_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $$1:$$2
replacement: '$1:$2'
target_label: __address__

# Maps all Kubernetes pod labels to Prometheus labels with the prefix removed (e.g., __meta_kubernetes_pod_label_app becomes app).
Expand Down Expand Up @@ -155,7 +156,7 @@ service:
receivers: [k8s_cluster]
processors: [memory_limiter, batch, k8sattributes, attributes/observe_common, attributes/debug_source_cluster_metrics]
exporters: [prometheusremotewrite, debug/override]
{{- if .Values.cluster.metrics.pod.enabled }}
{{- if .Values.cluster.metrics.prometheusScrape.enabled }}
metrics/pod_metrics:
receivers: [prometheus/pod_metrics]
processors: [memory_limiter, batch, k8sattributes, attributes/observe_common, attributes/debug_source_pod_metrics]
Expand Down
28 changes: 20 additions & 8 deletions charts/agent/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,20 @@ cluster:
enabled: true
metrics:
enabled: true
pod:
# use this option to scrape prometheus metrics from pods
prometheusScrape:
enabled: false
action: keep
interval: 10s
# add whatever namespace the collection is deployed to
namespace_drop_regex: (.*istio.*|.*ingress.*|kube-system|observe)
namespace_keep_regex: (.*)
port_keep_regex: .*metrics
metric_drop_regex: .*bucket
metric_keep_regex: (.*)
# namespaces to exclude from scraping
namespaceDropRegex: (.*istio.*|.*ingress.*|kube-system)
# namespaces to explicity include for scraping - can use or (ns1|ns2)
namespaceKeepRegex: (.*)
# port names to scrape from - can use or .*metrics|otherportname
portKeepRegex: .*metrics
# metrics to drop
metricDropRegex: .*bucket
# metrics to keep
metricKeepRegex: (.*)

namespaceOverride:
# ! This needs to have same value as namespaceOverride in deployments and daemonsets below
Expand Down Expand Up @@ -126,6 +130,8 @@ cluster-events:


podAnnotations: {
# This stops optional prometheus scrape config from picking up these pods
observeinc_com_scrape: 'false',
observe_monitor_purpose: observecollection,
observe_monitor_scrape: 'true',
observe_monitor_path: '/metrics',
Expand Down Expand Up @@ -259,6 +265,8 @@ cluster-metrics:
egressRules: [{}]

podAnnotations: {
# This stops optional prometheus scrape config from picking up these pods
observeinc_com_scrape: 'false',
observe_monitor_purpose: observecollection,
observe_monitor_scrape: 'true',
observe_monitor_path: '/metrics',
Expand Down Expand Up @@ -385,6 +393,8 @@ node-logs-metrics:
egressRules: [{}]

podAnnotations: {
# This stops optional prometheus scrape config from picking up these pods
observeinc_com_scrape: 'false',
observe_monitor_purpose: observecollection,
observe_monitor_scrape: 'true',
observe_monitor_path: '/metrics',
Expand Down Expand Up @@ -544,6 +554,8 @@ monitor:
egressRules: [{}]

podAnnotations: {
# This stops optional prometheus scrape config from picking up these pods
observeinc_com_scrape: 'false',
observe_monitor_purpose: observecollection,
observe_monitor_scrape: 'false',
observe_monitor_path: '/metrics',
Expand Down
37 changes: 35 additions & 2 deletions examples/agent/pod_metrics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,28 @@ kubectl apply -f sample-pod.yaml
kubectl apply -f sample-pod-no.yaml
kubectl apply -f sample-pod-prometheus-labels.yaml
kubectl apply -f sample-pod-prometheus-labels-no.yaml
```

## Port forward
You can curl a given service to increment counters from local machine by port forwarding.

These sevices are defined in sample-pod files
```
kubectl port-forward service/prometheus-example-app-service 8080:8080
kubectl port-forward service/prometheus-example-app-no-service 8080:8080
kubectl port-forward service/prometheus-example-app-promlabel-service 8080:8080
kubectl port-forward service/prometheus-example-app-promlabel-no-service 8080:8080
```

## Call service locally
If port forwarded you can use these curl commands
```
# Get metrics
curl localhost:8080/metrics
Expand All @@ -40,14 +52,35 @@ curl localhost:8080/err
kubectl delete -f sample-pod.yaml
kubectl delete -f sample-pod-no.yaml
kubectl delete -f sample-pod-prometheus-labels.yaml
kubectl delete -f sample-pod-prometheus-labels-no.yaml
```

## Deploy k8s monitoring with pod metrics collection enabled
The sample-pod.yaml is deployed in the default namespace unless you change the provided command.

It has annotations to change the metrics port to 8080 (observeinc_com_port: '8080') which will tell the scrape config to not use the default port of 8888.
It has annotations to change the metrics port, metrics path and that this pod should be scraped:
```
annotations:
observeinc_com_scrape: 'true'
observeinc_com_path: '/metrics'
observeinc_com_port: '8080'
```

The sample-pod-no.yaml sets observeinc_com_scrape: 'false' so no metrics should be collected from this pod.

Pods created by sample-pod-prometheus-labels and sample-pod-prometheus-labels-no yaml files use prometheus annotations to accomplish same functions as pods above:
```
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: '/metrics'
prometheus.io/port: '8080'
```

The pod-metrics-values.yaml file sets the namespace to scrape metrics from to default with namespaceKeepRegex and adds the web port name as valid with portKeepRegex.

The pod-metrics-values.yaml file sets the namespace to scrape metrics from to default with namespace_keep_regex and adds the web port name as valid with port_keep_regex.

```
helm install pod-metrics-example -n k8smonitoring \
Expand Down
17 changes: 13 additions & 4 deletions examples/agent/pod_metrics/pod-metrics-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,20 @@ cluster:
enabled: true
metrics:
enabled: true
pod:
# use this option to scrape prometheus metrics from pods
prometheusScrape:
enabled: true
namespace_keep_regex: (default)
metric_drop_regex: ""
port_keep_regex: .*metrics|web
interval: 10s
# namespaces to exclude from scraping
namespaceDropRegex: (.*istio.*|.*ingress.*|kube-system)
# namespaces to explicity include for scraping - can use or (ns1|ns2)
namespaceKeepRegex: (default)
# port names to scrape from - can use or .*metrics|otherportname
portKeepRegex: .*metrics|web
# metrics to drop
metricDropRegex: .*bucket
# metrics to keep
metricKeepRegex: (.*)

namespaceOverride:
value: k8smonitoring
Expand Down
73 changes: 73 additions & 0 deletions examples/agent/pod_metrics/sample-pod-prometheus-labels-no.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: prometheus-example-app-promlabel-no
name: prometheus-example-app-promlabel-no
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: prometheus-example-app-promlabel-no
template:
metadata:
labels:
app.kubernetes.io/name: prometheus-example-app-promlabel-no
annotations:
prometheus.io/scrape: false,
prometheus.io/path: '/metrics'
prometheus.io/port: '8080'
spec:
containers:
- name: prometheus-example-app-promlabel-no
image: quay.io/brancz/prometheus-example-app:v0.3.0
ports:
- name: web
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-example-app-promlabel-no-service
spec:
selector:
app.kubernetes.io/name: prometheus-example-app-promlabel-no
ports:
- protocol: TCP
port: 8080 # Exposed service port
targetPort: 8080
name: metrics
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: caller-cronjob
spec:
schedule: "*/1 * * * *" # Runs every minute
jobTemplate:
spec:
template:
spec:
containers:
- name: caller
image: curlimages/curl:latest # A lightweight curl image
env:
- name: SLEEP_TIME
value: "10" # Sleep time in seconds
- name: LOOP_COUNT
value: "36" # Number of iterations
command:
- /bin/sh
- -c
- |
for i in $(seq 1 $LOOP_COUNT); do
curl http://prometheus-example-app-promlabel-no-service:8080; # Adjust the URL and port as necessary
# Second call on even numbers
if [ $((i % 2)) -eq 0 ]; then
curl http://prometheus-example-app-promlabel-no-service:8080/err; # Second target service
echo "Second call on even #$i made."
fi
sleep $SLEEP_TIME;
done
restartPolicy: OnFailure
Loading

0 comments on commit d643670

Please sign in to comment.