Skip to content

Latest commit

 

History

History
29 lines (21 loc) · 2.4 KB

README.md

File metadata and controls

29 lines (21 loc) · 2.4 KB
Raw

Deploying with Kubernetes (k8s)

This directory contains a Kubernetes configuration file capable of provisioning a Data Prepper cluster.

Use kubectl to apply these configurations files with kubectl apply -f data-prepper-k8s.yaml

This will provision a scalable Data Prepper cluster capable of receiving trace data, however it is up to the user to decide how the service will be accessed by clients. A few examples include:

  1. Adding a load balancer with an ingress controller (for server-side load balancing)
  2. Exposing Kubernetes DNS so that clients can retrieve the addresses of Data Prepper pods (for client-side load balancing)
  3. Using something like ExternalDNS to synchronize an external DNS (for client-side load balancing)

File Contents

kind: ConfigMap

Contains a ConfigMap which provides pods the 2 configuration files required to launch a Data Prepper instance: pipelines.yaml and data-prepper-config.yaml.

pipelines.yaml

This file contains the standard Trace Analytics usecase pipeline configuration. A few adjustments are needed by the user:

  1. Replace the 2 stdout sinks with OpenSearch sinks to actually send data to OpenSearch.
  2. Optionally enable TLS by providing the necessary key files for the otel_trace_source plugin.

data-prepper-config.yaml

This file disables TLS for the Data Prepper service APIs (e.g. /metrics), so please adjust this if you wish to enable encryption in transit. It also provides peer forwarder configuration with dns discover mode. Optionally enable TLS and mTLS by providing the necessary certificate files for the peer forwarder. Find more information about core peer forwarder here.

kind: Service

Contains a headless Service which enables Data Prepper pods to discover others pods in the cluster peers via periodic DNS lookups. No changes are required.

kind: Deployment

Contains a Deployment which will create a single Data Prepper pod by default. No changes are required, however you can adjust the initial replica count if desired.