Identify when Vector has processed all the logs

[anupamdialpad]

Hi,

We are running Vector as a sidecar container inside K8s pods. Vector is ingesting log files from some locations and outputting it to some sinks. Is there a way to identify how much of the logs Vector has processed? Or has Vector processed all the logs and passed them to the sink?

[jdrouet]

Hi!
If you want to have those internal information and send them somewhere else, you can use the internal_metrics.
If you want to do it to troubleshoot, you can use vector top in the running container and you will see what vector is handling.
Tell me if you need more information 😉

[anupamdialpad]

Would definitely need more information 😄
With internal_metrics how can I check that Vector has processed all the log files (source) and has emitted the result out via the sink?

[jdrouet]

I opened this PR this morning to have a demo on how to use the internal_metrics with Vector and Prometheus.

You can see on the internal_metrics documentation that a component_received_events_total metric is accessible.
Depending on the sink you pick you can have the information of how many events went out.

For example, with the datadog_logs sink, you can have buffer_sent_events_total event that could fit your needs.

[anupamdialpad]

Programmatically can I get what is the count of "events in" for a given source using vector top? Would be great if you can add a code snippet for it.
Following is a sample vector config for a source I am using

[sources.log_source]
  type = "file"
  include = ["/var/log/somefile.log"]

  multiline.start_pattern = '.*'
  multiline.condition_pattern = '^(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2})'
  multiline.mode = "halt_before"
  multiline.timeout_ms = 1000

  ignore_older_secs = 86400 # 24h

[JeanMertz]

vector top is built on top of our GraphQL API that Vector exposes if you configure the top-level api property.

See here for more details: https://vector.dev/docs/reference/api/

You can use the GraphQL API to read the metrics presented in vector top.

Alternatively, you can subscribe to the internal_metrics source, and send those metrics to whatever external system you want to parse and act on the source-specific metrics.