Vector don't send logs from kubernetes cluster

[ZrytyADHD]

Hi,
I try to deploy Vector V0.44 on Tanzu Kubernetes Cluster.
I was able to do that using helm and application work also use this config below to test log sending to my external graylog:

[sources.demo_logs]
format = "syslog"
interval = 0.5
type = "demo_logs"

[sinks.graylog_syslog]
address = "graylog:12201"
inputs = ["demo_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "gelf"

And everything seems to work I can se this demo logs.
Problem is with kubernetes logs.

With this config in vector.toml:

[sources.kubernetes_logs]
type = "kubernetes_logs"
auto_partial_merge = true
delay_deletion_ms = 50

fingerprint_lines = 1
glob_minimum_cooldown_ms = 60000
ingestion_timestamp_field = ".ingest_timestamp"
max_line_bytes = 64768
max_read_bytes = 22048
oldest_first = true
read_from = "beginning"
self_node_name = "${VECTOR_SELF_NODE_NAME}"
timezone = "local"

logs are not sended to graylog, I think because they're not collected.

Can anyone help me or provide correct setting to collect them and sending very long logs like complete java stack traces?
Tanzu use containerd. With other collector like fluentbit or fluentd logs work perfectly.

[pront]

Do you have any sample log? Do you see any errors? Based on your sink configuration, you are trying to encode the log events with the gelf codec.

[ZrytyADHD]

Hi,
thanks for reply. I using gelf codec because logs are sended to Graylog which use it as input.
If I configure demo logs they are sended without problem even logs with more than 70k characters.

The problem is when I try collect them from my kubernetes cluster.
Example log from Velero below:
time="2025-01-21T09:13:30Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:142" time="2025-01-21T09:13:30Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:127" time="2025-01-21T09:13:30Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=234707 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws time="2025-01-21T09:14:06Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=234720 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws time="2025-01-21T09:14:30Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:142"

I can't see any errors in vector:

Container: vector
Connected
2025-01-21T21:06:12.373274Z  INFO vector::app: Log level is enabled. level="info"
2025-01-21T21:06:12.380710Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2025-01-21T21:06:12.384682Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="vector-0"
2025-01-21T21:06:12.400820Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Including matching files. ret=["**/*"]
2025-01-21T21:06:12.400924Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Excluding matching files. ret=["**/*.gz", "**/*.tmp"]
2025-01-21T21:06:12.402014Z  INFO vector::topology::running: Running healthchecks.
2025-01-21T21:06:12.402187Z  INFO vector: Vector has started. debug="false" version="0.44.0" arch="x86_64" revision="3cdc7c3 2025-01-13 21:26:04.735691656"
2025-01-21T21:06:12.402209Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2025-01-21T21:06:12.404511Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files.
2025-01-21T21:06:12.409574Z  INFO vector::topology::builder: Healthcheck passed.

In fluentbit I have this config:

 Name               tail
 Tag                kube.*
  Path               /var/log/containers/*.log
  Parser             docker
  DB                 /var/log/flb_graylog.db
    DB.Sync            Normal
    Docker_Mode        On
    Buffer_Chunk_Size  512KB
    Buffer_Max_Size    5M
    Rotate_Wait        30
    Mem_Buf_Limit      30MB
    Skip_Long_Lines    On
    Refresh_Interval   10

and everything work fine.

I choose to use vector because I need collect long logs from docker such as java starcktrace which fluentd and fluentbit can't concat together.

[pront]

So you can have plenty of options to help you diagnose here:

• vector tap to see outputs of your source (and hence what goes in to the sink) - guide here
• vector top to see sink metrics - docs here
• add an internal_metrics source and inspect the sink metrics

E.g.

sources:
  internal_metrics_1:
    type: internal_metrics
    scrape_interval_secs: 10

I would start by looking at https://vector.dev/docs/reference/configuration/sinks/socket/#component_sent_events_total and https://vector.dev/docs/reference/configuration/sinks/socket/#component_discarded_events_total.

Let me know if the above makes sense or if you have followup questions.

Tip: you can create blocks with three backticks (```)

[ZrytyADHD]

Thanks for answer I test some configurations and result below:

data_dir = "/var/lib/vector"

[api]
enabled = true
# vector.toml

[sources.kubernetes_logs]
  type = "internal_metrics"

[sinks.graylog_syslog]
address = "x.x.x.x:12216"
inputs = ["kubernetes_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "gelf"

Not working i got error:
2025-01-22T08:51:04.367060Z ERROR vector::cli: Configuration error. error=Data type mismatch between kubernetes_logs (["Metric"]) and graylog_syslog (["Log"])

This configration:

data_dir = "/var/lib/vector"

[api]
enabled = true

[sources.kubernetes_logs]
  type = "internal_metrics"

[sinks.graylog_syslog]
address = "x.x.x.x:12216"
inputs = ["kubernetes_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "text"

working and metrics are sended to graylog.

This one:

data_dir = "/var/lib/vector"

[api]
enabled = true
# vector.toml

[sources.kubernetes_logs]
  type = "file"
  include = ["/var/log/containers/*.log"]

[sinks.graylog_syslog]
address = "x.x.x.x:12216"
inputs = ["kubernetes_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "text"

not sending any logs. Changing codec from text to gelf etc. don't change anything.

Edit:
I test it on k3s cluster with the same effect logs are not sended to graylog. Both Tanzu and K3s uses containerd as runtime.

[jszwedko]

I think the above recommendations are still relevant:

• vector tap to see outputs of your source (and hence what goes in to the sink) - guide here
• vector top to see sink metrics - docs here
• add an internal_metrics source and inspect the sink metrics

It would be useful to use those to verify if:

• Vector is not consuming any logs
• Vector is consuming logs but the sink isn't sending them

In addition, could you share the log output from Vector? That should tell us if Vector is discovering the files at all. You mentioned running in a containerized environment. Have you mounted the /var/log/containers/ directory from the host into the Vector container?

[ZrytyADHD]

I test all of them:
vector top

vector tap

Adding internal metrics:

I archive some success changing config a little to file type:

[api]
enabled = true

[sources.kubernetes_logs]
  type = "file"
  include = ["/var/log/pods/*/*/*.log"]
  ignore_older = 86400
  read_from = "beginning"

[sinks.graylog_syslog]
address = "x.x.x.x:12216"
inputs = ["kubernetes_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "gelf"

With config above logs are sended to graylog:

I'm still looking for config to working as kuberntes_logs.

Log output from Vector with config:

[api]
enabled = true

[sources.kubernetes_logs]
  type = "kubernetes_logs"

[sinks.graylog_syslog]
address = "x.x.x.x:12216"
inputs = ["kubernetes_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "gelf"

Container: vector
Connected
2025-01-22T20:38:31.507601Z  INFO vector::app: Log level is enabled. level="info"
2025-01-22T20:38:31.510976Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2025-01-22T20:38:31.516380Z  INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name=""
2025-01-22T20:38:31.532700Z  INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Including matching files. ret=["**/*"]
2025-01-22T20:38:31.532740Z  INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Excluding matching files. ret=["**/*.gz", "**/*.tmp"]
2025-01-22T20:38:31.533777Z  INFO vector::topology::running: Running healthchecks.
2025-01-22T20:38:31.533924Z  INFO vector: Vector has started. debug="false" version="0.44.0" arch="x86_64" revision="3cdc7c3 2025-01-13 21:26:04.735691656"
2025-01-22T20:38:31.536392Z  INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files.
2025-01-22T20:38:31.539072Z  INFO vector::internal_events::api: API server running. address=127.0.0.1:8686 playground=http://127.0.0.1:8686/playground graphql=http://127.0.0.1:8686/graphql
2025-01-22T20:38:31.541485Z  INFO vector::topology::builder: Healthcheck passed. ```