How to increase through put for Vector consuming from kafka

[pkirubak]

Suspect two issues

1. Lua transform is reducing the consumption rate
2. with multiple instance of vector, the source is not getting load shared across replicas

Configuration:

sources:
  kafka_metric_read:
    type: kafka
    bootstrap_servers: “xxx.svc.cluster.local:9092"
    group_id: "vector-group"
    topics:
      - “t_metric"
    auto_offset_reset: "latest"
    acknowledgements:
      enabled: true

transforms:
  aggregator_metric_processing:
    type: remap
    inputs: ["kafka_metric_read"]
    source: |-
      .metric = parse_json!(.message)
      #.metric.timestamp = to_timestamp!(.metric.timestamp)

## refer https://github.com/vectordotdev/vector/discussions/7689#discussioncomment-885366
  metric_process:
    type: lua
    inputs:
      - aggregator_metric_processing
    version: '2'
    hooks:
      process: |
        function (event, emit)
          event.metric = event.log.metric
          event.log = nil
          emit(event)
        end

sinks:
  prom_metric:
    type: prometheus_remote_write
    inputs: ["metric_process"]
    endpoint: http://thanos-receive.svc.cluster.local:9598/api/v1/receive
    healthcheck: true
    request:
      retry_attempts: 0
      timeout_secs: 120

In kafka, the ingress rate is ~21k where as the egress is around 17k , so i suspect the bottle neck is not at transforms but at the vector's consuming from kafka ..

Looking at utilisation metrics , looks like there is some lag at lua transform.

sum(vector_utilization) by (component_type)

{component_type="remap"} | 0.0946328675065908
{component_type="lua"} | 0.9684126307364543
{component_type="prometheus_remote_write"} | 0.08025006493975331
{component_type="prometheus_exporter"} | 0.000161718097033596

Any suggestion on how to improve through put and how to enable multiple instance of vector to consume will help.

[jszwedko]

Hi @pkirubak !

We are rolling out a native (and native_json) codec that I think will allow you to avoid using the lua transform to convert JSON to metrics here which, as you note, appears to be the bottleneck. We expect this to be complete by v0.22.0.

For now, you could try splitting up the processing by partitioning the data and running multiple, identical, Lua transforms, that would run each in their own task. This would involve using the route transform with one route per parallel lua transform, partitioning across some field from the metric value that would fairly distribute (maybe the microsecond portion of the timestamp?), and then fanning back into the prom_metric sink. Let me know if that makes sense. I can try to sketch out more of an example if not.