Skip to content

Commit

Permalink
Merge branch 'main' into bring-back-check-format
Browse files Browse the repository at this point in the history
  • Loading branch information
trevorwhitney authored Mar 19, 2024
2 parents 8be0120 + 2e7dbe6 commit 575abe2
Show file tree
Hide file tree
Showing 52 changed files with 1,672 additions and 314 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

##### Enhancements

* [12243](https://github.com/grafana/loki/pull/12243) **lllamnyp**: Helm: Add extraContainers to read pods.
* [11840](https://github.com/grafana/loki/pull/11840) **jeschkies**: Allow custom usage trackers for ingested and discarded bytes metric.
* [11814](https://github.com/grafana/loki/pull/11814) **kavirajk**: feat: Support split align and caching for instant metric query results
* [11851](https://github.com/grafana/loki/pull/11851) **elcomtik**: Helm: Allow the definition of resources for GrafanaAgent pods.
Expand Down
10 changes: 9 additions & 1 deletion docs/sources/get-started/labels/structured-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,16 @@ Structured metadata is a way to attach metadata to logs without indexing them or
kubernetes pod names, process ID's, or any other label that is often used in queries but has high cardinality and is expensive
to extract at query time.

Structured metadata can also be used to query commonly needed metadata from log lines without needing to apply a parser at query time. Large json blobs or a poorly written query using complex regex patterns, for example, come with a high performance cost. Examples of useful metadata include trace IDs or user IDs.
Structured metadata can also be used to query commonly needed metadata from log lines without needing to apply a parser at query time. Large json blobs or a poorly written query using complex regex patterns, for example, come with a high performance cost. Examples of useful metadata include container_IDs or user IDs.

## When to use structured metadata

You should only use structured metadata in the following situations:

• If you are ingesting data in OpenTelemetry format, using the Grafana Agent or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data.
• If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include `process_id` or `thread_id` or Kubernetes pod names.

It is an antipattern to extract information that already exists in your log lines and put it into structured metadata.

## Attaching structured metadata to log lines

Expand Down
2 changes: 1 addition & 1 deletion docs/sources/operations/loki-canary/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ The `-labelname` and `-labelvalue` flags should also be provided, as these are
used by Loki Canary to filter the log stream to only process logs for the
current instance of the canary. Ensure that the values provided to the flags are
unique to each instance of Loki Canary. Grafana Labs' Tanka config
accomplishes this by passing in the pod name as the label value.
accomplishes this by passing in the Pod name as the label value.
If Loki Canary reports a high number of `unexpected_entries`, Loki Canary may
not be waiting long enough and the value for the `-wait` flag should be
Expand Down
14 changes: 8 additions & 6 deletions docs/sources/operations/storage/wal.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ The WAL also includes a backpressure mechanism to allow a large WAL to be replay

## Changes to deployment

1. Since ingesters need to have the same persistent volume across restarts/rollout, all the ingesters should be run on [statefulset](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) with fixed volumes.
1. Since ingesters need to have the same persistent volume across restarts/rollout, all the ingesters should be run on [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) with fixed volumes.

2. Following flags needs to be set
* `--ingester.wal-enabled` to `true` which enables writing to WAL during ingestion.
Expand All @@ -48,7 +48,7 @@ The WAL also includes a backpressure mechanism to allow a large WAL to be replay

## Changes in lifecycle when WAL is enabled

1. Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and brought back again with updated config. Hence flushing is skipped and the data is recovered from the WAL.
1. Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of StatefulSet there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and brought back again with updated config. Hence flushing is skipped and the data is recovered from the WAL.

## Disk space requirements

Expand All @@ -62,7 +62,7 @@ You should not target 100% disk utilisation.

## Migrating from stateless deployments

The ingester _deployment without WAL_ and _statefulset with WAL_ should be scaled down and up respectively in sync without transfer of data between them to ensure that any ingestion after migration is reliable immediately.
The ingester _Deployment without WAL_ and _StatefulSet with WAL_ should be scaled down and up respectively in sync without transfer of data between them to ensure that any ingestion after migration is reliable immediately.

Let's take an example of 4 ingesters. The migration would look something like this:

Expand All @@ -83,7 +83,7 @@ Scaling up is same as what you would do without WAL or statefulsets. Nothing to

When scaling down, we must ensure existing data on the leaving ingesters are flushed to storage instead of just the WAL. This is because we won't be replaying the WAL on an ingester that will no longer exist and we need to make sure the data is not orphaned.

Consider you have 4 ingesters `ingester-0 ingester-1 ingester-2 ingester-3` and you want to scale down to 2 ingesters, the ingesters which will be shutdown according to statefulset rules are `ingester-3` and then `ingester-2`.
Consider you have 4 ingesters `ingester-0 ingester-1 ingester-2 ingester-3` and you want to scale down to 2 ingesters, the ingesters which will be shut down according to StatefulSet rules are `ingester-3` and then `ingester-2`.

Hence before actually scaling down in Kubernetes, port forward those ingesters and hit the [`/ingester/shutdown?flush=true`]({{< relref "../../reference/api#flush-in-memory-chunks-and-shut-down" >}}) endpoint. This will flush the chunks and remove itself from the ring, after which it will register as unready and may be deleted.

Expand All @@ -95,13 +95,15 @@ After hitting the endpoint for `ingester-2 ingester-3`, scale down the ingesters

Statefulsets are significantly more cumbersome to work with, upgrade, and so on. Much of this stems from immutable fields on the specification. For example, if one wants to start using the WAL with single store Loki and wants separate volume mounts for the WAL and the boltdb-shipper, you may see immutability errors when attempting updates the Kubernetes statefulsets.

In this case, try `kubectl -n <namespace> delete sts ingester --cascade=false`. This will leave the pods alive but delete the statefulset. Then you may recreate the (updated) statefulset and one-by-one start deleting the `ingester-0` through `ingester-n` pods _in that order_, allowing the statefulset to spin up new pods to replace them.
In this case, try `kubectl -n <namespace> delete sts ingester --cascade=false`.
This will leave the Pods alive but delete the StatefulSet.
Then you may recreate the (updated) StatefulSet and one-by-one start deleting the `ingester-0` through `ingester-n` Pods _in that order_, allowing the StatefulSet to spin up new pods to replace them.

#### Scaling Down Using `/flush_shutdown` Endpoint and Lifecycle Hook

1. **StatefulSets for Ordered Scaling Down**: Loki's ingesters should be scaled down one by one, which is efficiently handled by Kubernetes StatefulSets. This ensures an ordered and reliable scaling process, as described in the [Deployment and Scaling Guarantees](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees) documentation.

2. **Using PreStop Lifecycle Hook**: During the pod scaling down process, the PreStop [lifecycle hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) triggers the `/flush_shutdown` endpoint on the ingester. This action flushes the chunks and removes the ingester from the ring, allowing it to register as unready and become eligible for deletion.
2. **Using PreStop Lifecycle Hook**: During the Pod scaling down process, the PreStop [lifecycle hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) triggers the `/flush_shutdown` endpoint on the ingester. This action flushes the chunks and removes the ingester from the ring, allowing it to register as unready and become eligible for deletion.

3. **Using terminationGracePeriodSeconds**: Provides time for the ingester to flush its data before being deleted, if flushing data takes more than 30 minutes, you may need to increase it.

Expand Down
16 changes: 8 additions & 8 deletions docs/sources/operations/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,14 +123,14 @@ promtail -log.level=debug
The Promtail configuration contains a `__path__` entry to a directory that
Promtail cannot find.

## Connecting to a Promtail pod to troubleshoot
## Connecting to a Promtail Pod to troubleshoot

First check [Troubleshooting targets](#troubleshooting-targets) section above.
If that doesn't help answer your questions, you can connect to the Promtail pod
If that doesn't help answer your questions, you can connect to the Promtail Pod
to investigate further.

If you are running Promtail as a DaemonSet in your cluster, you will have a
Promtail pod on each node, so figure out which Promtail you need to debug first:
Promtail Pod on each node, so figure out which Promtail you need to debug first:


```shell
Expand All @@ -145,10 +145,10 @@ promtail-bth9q 1/1 Running 0 3h 10.56.
That output is truncated to highlight just the two pods we are interested in,
you can see with the `-o wide` flag the NODE on which they are running.

You'll want to match the node for the pod you are interested in, in this example
You'll want to match the node for the Pod you are interested in, in this example
NGINX, to the Promtail running on the same node.

To debug you can connect to the Promtail pod:
To debug you can connect to the Promtail Pod:

```shell
kubectl exec -it promtail-bth9q -- /bin/sh
Expand Down Expand Up @@ -182,12 +182,12 @@ $ helm upgrade --install loki loki/loki --set "loki.tracing.enabled=true"

## Running Loki with Istio Sidecars

An Istio sidecar runs alongside a pod. It intercepts all traffic to and from the pod.
When a pod tries to communicate with another pod using a given protocol, Istio inspects the destination's service using [Protocol Selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/).
An Istio sidecar runs alongside a Pod. It intercepts all traffic to and from the Pod.
When a Pod tries to communicate with another Pod using a given protocol, Istio inspects the destination's service using [Protocol Selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/).
This mechanism uses a convention on the port name (for example, `http-my-port` or `grpc-my-port`)
to determine how to handle this outgoing traffic. Istio can then do operations such as authorization and smart routing.

This works fine when one pod communicates with another pod using a hostname. But,
This works fine when one Pod communicates with another Pod using a hostname. But,
Istio does not allow pods to communicate with other pods using IP addresses,
unless the traffic type is `tcp`.

Expand Down
2 changes: 1 addition & 1 deletion docs/sources/operations/zone-ingesters.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ These instructions assume you are using the zone aware ingester jsonnet deployme

1. if you're using an automated reconcilliation/deployment system like flux, disable it now (for example using flux ignore), if possible for just the default ingester StatefulSet

1. Shutdown flush the default ingesters, unregistering them from the ring, you can do this by port-forwarding each ingester pod and using the endpoint: `"http://url:PORT/ingester/shutdown?flush=true&delete_ring_tokens=true&terminate=false"`
1. Shutdown flush the default ingesters, unregistering them from the ring, you can do this by port-forwarding each ingester Pod and using the endpoint: `"http://url:PORT/ingester/shutdown?flush=true&delete_ring_tokens=true&terminate=false"`

1. manually scale down the default ingester StatefulSet to 0 replicas, we do this via `tk apply` but you could do it via modifying the yaml

Expand Down
9 changes: 9 additions & 0 deletions docs/sources/setup/install/helm/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -3345,6 +3345,15 @@ null
<td><pre lang="json">
[]
</pre>
</td>
</tr>
<tr>
<td>read.extraContainers</td>
<td>list</td>
<td>Containers to add to the read pods</td>
<td><pre lang="json">
[]
</pre>
</td>
</tr>
<tr>
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ require (
github.com/IBM/ibm-cos-sdk-go v1.10.0
github.com/axiomhq/hyperloglog v0.0.0-20240124082744-24bca3a5b39b
github.com/d4l3k/messagediff v1.2.1
github.com/dolthub/swiss v0.2.1
github.com/efficientgo/core v1.0.0-rc.2
github.com/fsnotify/fsnotify v1.6.0
github.com/gogo/googleapis v1.4.0
Expand Down Expand Up @@ -199,6 +200,7 @@ require (
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-metrics v0.0.1 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dolthub/maphash v0.1.0 // indirect
github.com/eapache/go-resiliency v1.3.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230111030713-bf00bc1b83b6 // indirect
github.com/eapache/queue v1.1.0 // indirect
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,10 @@ github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docker/libnetwork v0.8.0-dev.2.0.20181012153825-d7b61745d166/go.mod h1:93m0aTqz6z+g32wla4l4WxTrdtvBRmVzYRkYvasA5Z8=
github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
github.com/dolthub/maphash v0.1.0 h1:bsQ7JsF4FkkWyrP3oCnFJgrCUAFbFf3kOl4L/QxPDyQ=
github.com/dolthub/maphash v0.1.0/go.mod h1:gkg4Ch4CdCDu5h6PMriVLawB7koZ+5ijb9puGMV50a4=
github.com/dolthub/swiss v0.2.1 h1:gs2osYs5SJkAaH5/ggVJqXQxRXtWshF6uE0lgR/Y3Gw=
github.com/dolthub/swiss v0.2.1/go.mod h1:8AhKZZ1HK7g18j7v7k6c5cYIGEZJcPn0ARsai8cUrh0=
github.com/drone/envsubst v1.0.3 h1:PCIBwNDYjs50AsLZPYdfhSATKaRg/FJmDc2D6+C2x8g=
github.com/drone/envsubst v1.0.3/go.mod h1:N2jZmlMufstn1KEqvbHjw40h1KyTmnVzHcSc9bFiJ2g=
github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
Expand Down
1 change: 1 addition & 0 deletions operator/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
## Main

- [12228](https://github.com/grafana/loki/pull/12228) **xperimental**: Restructure LokiStack metrics
- [12164](https://github.com/grafana/loki/pull/12164) **periklis**: Use safe bearer token authentication to scrape operator metrics
- [12216](https://github.com/grafana/loki/pull/12216) **xperimental**: Fix duplicate operator metrics due to ServiceMonitor selector
- [12212](https://github.com/grafana/loki/pull/12212) **xperimental**: Keep credentialMode in status when updating schemas
Expand Down
7 changes: 0 additions & 7 deletions operator/internal/handlers/lokistack_create_or_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ import (
"github.com/grafana/loki/operator/internal/handlers/internal/storage"
"github.com/grafana/loki/operator/internal/handlers/internal/tlsprofile"
"github.com/grafana/loki/operator/internal/manifests"
"github.com/grafana/loki/operator/internal/metrics"
"github.com/grafana/loki/operator/internal/status"
)

Expand Down Expand Up @@ -208,12 +207,6 @@ func CreateOrUpdateLokiStack(
return "", kverrors.New("failed to configure lokistack resources", "name", req.NamespacedName)
}

// 1x.demo is used only for development, so the metrics will not
// be collected.
if opts.Stack.Size != lokiv1.SizeOneXDemo {
metrics.Collect(&opts.Stack, opts.Name)
}

return objStore.CredentialMode, nil
}

Expand Down
14 changes: 10 additions & 4 deletions operator/internal/manifests/internal/alerts/prometheus-alerts.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -178,10 +178,16 @@ groups:
- alert: LokistackSchemaUpgradesRequired
annotations:
message: |-
Object storage schema needs upgrade.
summary: "The applied storage schema config is old and should be upgraded."
The LokiStack "{{ $labels.stack_name }}" in namespace "{{ $labels.stack_namespace }}" is using a storage schema
configuration that does not contain the latest schema version. It is recommended to update the schema
configuration to update the schema version to the latest version in the future.
summary: "One or more of the deployed LokiStacks contains an outdated storage schema configuration."
runbook_url: "[[ .RunbookURL ]]#Lokistack-Schema-Upgrades-Required"
expr: sum by(stack_id) (lokistack_warnings_count) > 0
expr: |
sum (
lokistack_status_condition{reason="StorageNeedsSchemaUpdate",status="true"}
) by (stack_namespace, stack_name)
> 0
for: 1m
labels:
severity: warning
resource: '{{ $labels.stack_id}}'
96 changes: 96 additions & 0 deletions operator/internal/metrics/lokistack.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
package metrics

import (
"context"

"github.com/go-logr/logr"
"github.com/prometheus/client_golang/prometheus"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"

lokiv1 "github.com/grafana/loki/operator/apis/loki/v1"
)

const (
metricsPrefix = "lokistack_"
)

var (
metricsCommonLabels = []string{
"stack_namespace",
"stack_name",
"size",
}

lokiStackInfoDesc = prometheus.NewDesc(
metricsPrefix+"info",
"Information about deployed LokiStack instances. Value is always 1.",
metricsCommonLabels, nil,
)

lokiStackConditionsCountDesc = prometheus.NewDesc(
metricsPrefix+"status_condition",
"Counts the current status conditions of the LokiStack.",
append(metricsCommonLabels, "condition", "reason", "status"), nil,
)
)

func RegisterLokiStackCollector(log logr.Logger, k8sClient client.Client, registry prometheus.Registerer) error {
metrics := &lokiStackCollector{
log: log,
k8sClient: k8sClient,
}

return registry.Register(metrics)
}

type lokiStackCollector struct {
log logr.Logger
k8sClient client.Client
}

func (l *lokiStackCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- lokiStackInfoDesc
ch <- lokiStackConditionsCountDesc
}

func (l *lokiStackCollector) Collect(m chan<- prometheus.Metric) {
ctx := context.TODO()

stackList := &lokiv1.LokiStackList{}
err := l.k8sClient.List(ctx, stackList)
if err != nil {
l.log.Error(err, "failed to get list of LokiStacks for metrics")
return
}

for _, stack := range stackList.Items {
labels := []string{
stack.Namespace,
stack.Name,
string(stack.Spec.Size),
}

m <- prometheus.MustNewConstMetric(lokiStackInfoDesc, prometheus.GaugeValue, 1.0, labels...)

for _, c := range stack.Status.Conditions {
activeValue := 0.0
if c.Status == metav1.ConditionTrue {
activeValue = 1.0
}

// This mirrors the behavior of kube_state_metrics, which creates two metrics for each condition,
// one for each status (true/false).
m <- prometheus.MustNewConstMetric(
lokiStackConditionsCountDesc,
prometheus.GaugeValue, activeValue,
append(labels, c.Type, c.Reason, "true")...,
)
m <- prometheus.MustNewConstMetric(
lokiStackConditionsCountDesc,
prometheus.GaugeValue, 1.0-activeValue,
append(labels, c.Type, c.Reason, "false")...,
)
}
}
}
Loading

0 comments on commit 575abe2

Please sign in to comment.