Topics mirrored by MirrorMaker2 are replicated with much larger size

[eslam-gomaa]

In my case I'm replicating topics from an old cluster (confluent) to a new cluster (strimzi) with MirrorMaker2

I got "No space left on device" error on the new cluster, and as part of investigation I found that some of the replicated topics are replicated with much larger size

Example:

Confluent cluster

# du -h /opt/kafka/data-0/logs/ | grep el-server-temp
11G     /opt/kafka/data-0/logs/el-server-temp-0

Strimzi cluster

# du -h /var/lib/kafka/data-0/kafka-log1 | grep el-server-temp
89G     /var/lib/kafka/data-0/kafka-log1/el-server-temp-0

NOTE: both clusters have the same retention period of 7 days (the defaults)

---

  clusters:
  - alias: "source-cluster"
    bootstrapServers: ......
  - alias: "destination-cluster"
    bootstrapServers: ......
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  # Mirror topics from source cluster (old) to destination cluster (new)
  - sourceCluster: "source-cluster"
    targetCluster: "destination-cluster"
    sourceConnector:
      config:
#        producer.override.batch.size: 327680
        producer.override.linger.ms: 100
        producer.request.timeout.ms: 30000
        consumer.fetch.max.bytes: 52428800
        # The following 2 config options are for (mirroring the topics without cluster name prefix)
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        replication.policy.separator: ""
        replication.factor: -1
        offset-syncs.topic.replication.factor: 3
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        producer.override.request.timeout.ms: 30000
        heartbeats.topic.replication.factor:  3
    checkpointConnector:
      config:
        # The following 2 config options are important (to sync the conumer groups offsets)
        sync.group.offsets.enabled: true
        emit.checkpoints.enabled: true
        producer.override.request.timeout.ms: 30000
        consumer.max.poll.interval.ms: 300000
        checkpoints.topic.replication.factor: 3
    topicsPattern: ".*"
    groupsPattern: ".*"
    # Topics to execlude
    topicsExcludePattern: ".*confluent.*"

---

MirrorMaker2 logs

@scholzj Thanks to let me know if there is something wrong in my configurations

[scholzj]

I have no clue, sorry. I'm not an MM2 expert and I also don't know the settings on the clusters. I think you need to compare the folders and the topic configurations and check what differs.

[eslam-gomaa]

@scholzj I have checked the config and they're similar in both clusters

I actually did many test cases but most important are these two:

Background:

• the problem happens with me when I'm mirroring large topics (with hundreds of millions of messages of messages, eg. 670m message/topic)
• I'm trying to mirror large topics from a 2.8 Kafka cluster to 3.2 cluster

1. When mirrored topics from a cluster (dev-cluster) with few messages (< 100 million messages total) the issue didn't appear, but the issue happend when mirroring from a cluster (prod cluster) with large topics (> 10 billion messages total )

2. tested Mirroring data between 2 strimzi clusters with the same version (3.2) and didn't see the issue

I just need to know if that behaviour is expected when using MirrorMaker2 with different Kafka versions ?

---

Here is the config

New cluster config

kafkaCluster:
  version: 3.2.0
  interBrokerProtocolVersion: "3.2"

  kafka:
    replicas: 3
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: true
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      auto.create.topics.enable: false
      delete.topic.enable: true
      num.io.threads: 8
      num.recovery.threads.per.data.dir: 1
      unclean.leader.election.enable: false
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      # set min.insync.replicas to 1, because MirrorMaker is configured with replicaiton factor of 1 (for testing)
      min.insync.replicas: 1
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 500Gi
        class: gp2
        deleteClaim: false
    resources:
      requests:
        cpu: 1
        memory: 8Gi
      limits:
        cpu: 2
        memory: 20Gi
    jvmOptions:
      -Xms: 4G
      -Xmx: 4G

  zookeeper:
    replicas: 3
    template:
      # We need to add Prometheus annotations, so that we can auto discover the Pod metrics by Prometheus (if needed)
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: true
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
    storage:
      type: persistent-claim
      size: 200Gi
      class: gp2
      deleteClaim: false
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 2
        memory: 2Gi
    entityOperator:
      topicOperator:
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 1
            memory: 256Mi
      userOperator:
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 1
            memory: 512Mi

  kafkaExporter:
    template:
      # We need to add Prometheus annotations, so that we can auto discover the Pod metrics by Prometheus (if needed)
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: true
            prometheus.io/path: /metrics
            prometheus.io/port: tcp-prometheus
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

broker config for both clusters

source cluster: 2.8-config.txt
destination cluster: 3.2-config.txt

[celestialsahil]

Any update on this? I am facing similar issue with target disk size increase while using mm2.