What's the difference between the two parameters: "producer_batch_num" of kafka-logger and "batch_max_size" of batch-processor?

[mangolost]

What's the difference between the two parameters: "producer_batch_num" of kafka-logger and "batch_max_size" of batch-processor?

I would like to use kafka-logger to record http requests into kafka. In order to decrease the commit frequency, it is suggested to send batch records to kafka other than sending records one by one. I have read the document and it makes me confused.

https://apisix.apache.org/docs/apisix/3.2/plugins/kafka-logger/

https://apisix.apache.org/docs/apisix/3.2/batch-processor/

How should I config my settings to send batched records to kafka?

[dosubot[bot]]

Hey @mangolost! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

To configure your settings for sending batched records to Kafka using the kafka-logger plugin and the batch-processor plugin, you need to understand the roles of the different parameters:

1. producer_batch_num (kafka-logger): Specifies the number of messages to be merged and sent to the Kafka server in a single batch. This controls how many individual messages are grouped together before being sent.

2. producer_batch_size (kafka-logger): Defines the maximum size of the batch in bytes. This limits the total size of the messages that can be sent in one batch.

3. batch_max_size (batch-processor): Specifies the maximum number of logs or entries that can be processed in a single batch before they are sent to the next stage in the processing pipeline. The default value is 1000 ^[1][2].

Configuration Steps:

1. Set producer_batch_num:

   • Determine the number of messages you want to include in each batch sent to Kafka.
   • Example: producer_batch_num: 200

2. Set producer_batch_size:

   • Define the maximum size of each batch in bytes.
   • Example: producer_batch_size: 1048576 (1 MB)

3. Set batch_max_size:

   • Configure the batch_max_size in the batch-processor to control how many entries are processed in a single batch.
   • Example: batch_max_size: 1000

Example Configuration:

kafka-logger:
  producer_batch_num: 200
  producer_batch_size: 1048576

batch-processor:
  batch_max_size: 1000

By setting these parameters, you can control the batching behavior to reduce the commit frequency to Kafka while efficiently processing logs.

To continue talking to Dosu, mention @dosu.