From b84d5a5d4f20e13e85746b88c435e86def712f28 Mon Sep 17 00:00:00 2001 From: prmellor Date: Tue, 1 Oct 2024 16:52:59 +0100 Subject: [PATCH 1/3] docs(connectors): adds procedures to manage connector offsets Signed-off-by: prmellor --- .../configuring/assembly-config.adoc | 6 + .../con-config-mirrormaker2-connectors.adoc | 33 +++- .../proc-altering-connector-offsets.adoc | 127 +++++++++++++++ .../proc-listing-connector-offsets.adoc | 146 ++++++++++++++++++ .../proc-manual-restart-connector-task.adoc | 2 +- .../proc-manual-restart-connector.adoc | 2 +- .../proc-manual-stop-pause-connector.adoc | 2 +- .../proc-resetting-connector-offsets.adoc | 87 +++++++++++ ...-oauth-authorization-keycloak-example.adoc | 2 +- .../modules/upgrading/con-upgrade-status.adoc | 2 +- 10 files changed, 403 insertions(+), 6 deletions(-) create mode 100644 documentation/modules/configuring/proc-altering-connector-offsets.adoc create mode 100644 documentation/modules/configuring/proc-listing-connector-offsets.adoc create mode 100644 documentation/modules/configuring/proc-resetting-connector-offsets.adoc diff --git a/documentation/assemblies/configuring/assembly-config.adoc b/documentation/assemblies/configuring/assembly-config.adoc index 731ad489b0d..410d35998dd 100644 --- a/documentation/assemblies/configuring/assembly-config.adoc +++ b/documentation/assemblies/configuring/assembly-config.adoc @@ -135,6 +135,12 @@ include::../../modules/configuring/proc-manual-stop-pause-connector.adoc[levelof include::../../modules/configuring/proc-manual-restart-connector.adoc[leveloffset=+2] //Procedure to manually restart a Kafka connector task include::../../modules/configuring/proc-manual-restart-connector-task.adoc[leveloffset=+2] +//procedure to list offsets +include::../../modules/configuring/proc-listing-connector-offsets.adoc[leveloffset=+2] +//procedure to alter offsets +include::../../modules/configuring/proc-altering-connector-offsets.adoc[leveloffset=+2] +//procedure to reset offsets +include::../../modules/configuring/proc-resetting-connector-offsets.adoc[leveloffset=+2] //`KafkaMirrorMaker2` resource config include::../../modules/configuring/con-config-mirrormaker2.adoc[leveloffset=+1] diff --git a/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc b/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc index 5199b8a146d..18717953c8a 100644 --- a/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc +++ b/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc @@ -242,4 +242,35 @@ Specifically, ensure that the following properties have the same value across al * `topic.filter.class` For example, the value for `replication.policy.class` must be the same for the source, checkpoint, and heartbeat connectors. -Mismatched or missing settings cause issues with data replication or offset syncing, so it's essential to keep all relevant connectors configured with the same settings. \ No newline at end of file +Mismatched or missing settings cause issues with data replication or offset syncing, so it's essential to keep all relevant connectors configured with the same settings. + +== Listing the offsets of MirrorMaker 2 connectors + +To list the offset positions of the internal MirrorMaker 2 connectors, use the same configuration that's used to manage Kafka Connect connectors. + +In this example, the `sourceConnector` configuration is updated to return the connector offset position. +The offset information is written to a specified config map. + +.Example configuration for connector producers and consumers +[source,yaml,subs="+quotes,attributes"] +---- +apiVersion: {KafkaMirrorMaker2ApiVersion} +kind: KafkaMirrorMaker2 +metadata: + name: my-mirror-maker2 +spec: + version: {DefaultKafkaVersion} + # ... + mirrors: + - sourceCluster: "my-cluster-source" + targetCluster: "my-cluster-target" + sourceConnector: + listOffsets: + toConfigMap: + name: my-connector-offsets + # ... +---- + +For more information on setting up the configuration, see xref:proc-listing-connector-offsets-{context}[]. + +NOTE: It is possible to use configuration to xref:proc-altering-connector-offsets-{context}[alter] or xref:proc-resetting-connector-offsets-{context}[reset] connector offsets, though this is rarely necessary. \ No newline at end of file diff --git a/documentation/modules/configuring/proc-altering-connector-offsets.adoc b/documentation/modules/configuring/proc-altering-connector-offsets.adoc new file mode 100644 index 00000000000..c16683a1ecc --- /dev/null +++ b/documentation/modules/configuring/proc-altering-connector-offsets.adoc @@ -0,0 +1,127 @@ +// Module included in the following assemblies: +// +// assembly-config.adoc + +[id='proc-altering-connector-offsets-{context}'] += Altering connector offsets + +[role="_abstract"] +To alter connector offsets using `KafkaConnector` resources, add the `alterOffsets` configuration. +This can be useful, for example, to skip a _poison_ record or replay a record. +The offset changes are specified in a config map. +You can reuse the same config map used to xref:proc-listing-connector-offsets-{context}[list offsets]. +The offset alteration is applied by annotating the `KafkaConnector` resource. + +In this procedure, we alter the offset position for a source connector named `my-source-connector`. + +NOTE: You can also alter connector offsets using the `PATCH /connectors/{connector}/offsets` request through the Kafka Connect Rest API. + +.Prerequisites + +* The Cluster Operator is running. + +.Procedure + +. Edit the `KafkaConnector` resource to stop the connector and include the `listOffsets` configuration. ++ +.Example configuration to stop a connector and alter offsets +[source,yaml,subs="+attributes"] +---- +apiVersion: {KafkaConnectorApiVersion} +kind: KafkaConnector +metadata: + name: my-source-connector + labels: + strimzi.io/cluster: my-connect-cluster +spec: + state: stopped # <1> + alterOffsets: + fromConfigMap: # <2> + name: my-connector-offsets # <3> + # ... +---- +<1> Changes the state of the connector to `stopped`. The default state for the connector when this property is not set is `running`. +<2> The reference to the config map that provides the update. +<3> The name of the config map, which is named `my-connector-offsets` in this example. + +. Edit the config map to make the alteration. ++ +In this example, we're resetting the offset position for a source connector to 15000. ++ +.Example source connector offset list configuration +[source,yaml,subs="+attributes"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + # ... +data: + offsets.json: |- + { + "offsets" : [ { + "partition" : { + "filename" : "/data/myfile.txt" + }, + "offset" : { + "position" : 15000 # <1> + } + } ] + } +---- +<1> The updated offset position in the source partition. + +. Run the command to update the offset position by annotating the `KafkaConnector` resource: ++ +[source,shell] +---- +kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=alter -n +---- ++ +The annotation remains until either the update operation succeeds, or it is manually removed from the resource. + +. Inspect the contents of the config map to verify the update to the offset position: ++ +[source,shell] +---- +kubectl describe configmap my-connector-offsets -n +---- ++ +Strimzi puts the offsets into the `offsets.json` property. +It doesn't overwrite any other properties when updating an existing config map. ++ +.Example source connector offset list +[source,yaml,subs="+attributes"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + # ... +data: + offsets.json: |- + { + "offsets" : [ { + "partition" : { + "filename" : "/data/myfile.txt" + }, + "offset" : { + "position" : 15000 + } + } ] + } +---- + +. Restart the connector by changing the state to `running`. ++ +.Example configuration to start a connector +[source,yaml,subs="+attributes"] +---- +apiVersion: {KafkaConnectorApiVersion} +kind: KafkaConnector +metadata: + name: my-source-connector + labels: + strimzi.io/cluster: my-connect-cluster +spec: + state: running + # ... +---- \ No newline at end of file diff --git a/documentation/modules/configuring/proc-listing-connector-offsets.adoc b/documentation/modules/configuring/proc-listing-connector-offsets.adoc new file mode 100644 index 00000000000..478fcc6dfcd --- /dev/null +++ b/documentation/modules/configuring/proc-listing-connector-offsets.adoc @@ -0,0 +1,146 @@ +// Module included in the following assemblies: +// +// assembly-config.adoc + +[id='proc-listing-connector-offsets-{context}'] += Listing connector offsets + +[role="_abstract"] +To track connector offsets using `KafkaConnector` resources, add the `listOffsets` configuration. +The offsets, which keep track of the flow of data, are written to a config map specified in the configuration. +If the config map does not exist, Strimzi creates it. +The list is written to the config map by annotating the `KafkaConnector` resource. + +Sink connectors use Kafka's standard consumer offset mechanism, while source connectors store offsets in a custom format within a Kafka topic. + +* For sink connectors, the list shows Kafka topic partitions and the last committed offset for each partition. +* For source connectors, the list shows the source system’s partition and the last offset processed. + +NOTE: You can also list connector offsets using the `GET /connectors/{connector}/offsets` request through the Kafka Connect Rest API. + +.Prerequisites + +* The Cluster Operator is running. + +.Procedure + +. Edit the `KafkaConnector` resource for the connector to include the `listOffsets` configuration. ++ +.Example configuration to list offsets +[source,yaml,subs="+attributes"] +---- +apiVersion: {KafkaConnectorApiVersion} +kind: KafkaConnector +metadata: + name: my-source-connector + labels: + strimzi.io/cluster: my-connect-cluster +spec: + listOffsets: + toConfigMap: # <1> + name: my-connector-offsets # <2> + # ... +---- +<1> The reference to the config map where the list of offsets will be written to. +<2> The name of the config map, which is named `my-connector-offsets` in this example. + +. Run the command to write the list to the config map by annotating the `KafkaConnector` resource: ++ +[source,shell] +---- +kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=list -n +---- ++ +The annotation remains until either the list operation succeeds, or it is manually removed from the resource. + +. After the `KafkaConnector` resource is updated, use the following command to check if the config map with the offsets was created: ++ +[source,shell] +---- +kubectl get configmap my-connector-offsets -n +---- + +. Inspect the contents of the config map to verify the offsets are being listed: ++ +[source,shell] +---- +kubectl describe configmap my-connector-offsets -n +---- ++ +Strimzi puts the offsets into the `offsets.json` property. +It doesn't overwrite any other properties when updating an existing config map. ++ +-- +.Example source connector offset list +[source,yaml,subs="+attributes"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + # ... + ownerReferences: # <1> + - apiVersion: {KafkaConnectApiVersion} + blockOwnerDeletion: false + controller: false + kind: KafkaConnector + name: my-source-connector + uid: 637e3be7-bd96-43ab-abde-c55b4c4550e0 + resourceVersion: "66951" + uid: 641d60a9-36eb-4f29-9895-8f2c1eb9638e +data: + offsets.json: |- + { + "offsets" : [ { + "partition" : { + "filename" : "/data/myfile.txt" # <2> + }, + "offset" : { + "position" : 15295 # <3> + } + } ] + } +---- +<1> The owner reference pointing to the `KafkaConnector` resource for the source connector. +To provide a custom owner reference, create the config map in advance and set the owner reference. +<2> The source partition, represented by the filename `/data/myfile.txt` in this example for a file-based connector. +<3> The last processed offset position in the source partition. +-- ++ +-- +.Example sink connector offset list +[source,yaml,subs="+attributes"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + # ... + ownerReferences: # <1> + - apiVersion: {KafkaConnectApiVersion} + blockOwnerDeletion: false + controller: false + kind: KafkaConnector + name: my-sink-connector + uid: 84a29d7f-77e6-43ac-bfbb-719f9b9a4b3b + resourceVersion: "79241" + uid: 721e30bc-23df-41a2-9b48-fb2b7d9b042c +data: + offsets.json: |- + { + "offsets": [ + { + "partition": { + "kafka_topic": "my-topic", # <2> + "kafka_partition": 2 # <3> + }, + "offset": { + "kafka_offset": 4 # <4> + } + } + ] + } +---- +<1> The owner reference pointing to the `KafkaConnector` resource for the sink connector. +<2> The Kafka topic that the sink connector is consuming from. +<3> The partition of the Kafka topic. +<4> The last committed Kafka offset for this topic and partition. +-- \ No newline at end of file diff --git a/documentation/modules/configuring/proc-manual-restart-connector-task.adoc b/documentation/modules/configuring/proc-manual-restart-connector-task.adoc index d6a116ddf4e..1c167651596 100644 --- a/documentation/modules/configuring/proc-manual-restart-connector-task.adoc +++ b/documentation/modules/configuring/proc-manual-restart-connector-task.adoc @@ -1,6 +1,6 @@ // Module included in the following assemblies: // -// assembly-deploy-kafka-connect-with-plugins.adoc +// assembly-config.adoc [id='proc-manual-restart-connector-task-{context}'] = Manually restarting Kafka Connect connector tasks diff --git a/documentation/modules/configuring/proc-manual-restart-connector.adoc b/documentation/modules/configuring/proc-manual-restart-connector.adoc index 9be4081b12a..ad6e413602f 100644 --- a/documentation/modules/configuring/proc-manual-restart-connector.adoc +++ b/documentation/modules/configuring/proc-manual-restart-connector.adoc @@ -1,6 +1,6 @@ // Module included in the following assemblies: // -// assembly-deploy-kafka-connect-with-plugins.adoc +// assembly-config.adoc [id='proc-manual-restart-connector-{context}'] = Manually restarting Kafka Connect connectors diff --git a/documentation/modules/configuring/proc-manual-stop-pause-connector.adoc b/documentation/modules/configuring/proc-manual-stop-pause-connector.adoc index b7da9fa96e8..7c83a4dd150 100644 --- a/documentation/modules/configuring/proc-manual-stop-pause-connector.adoc +++ b/documentation/modules/configuring/proc-manual-stop-pause-connector.adoc @@ -1,6 +1,6 @@ // Module included in the following assemblies: // -// assembly-deploy-kafka-connect-with-plugins.adoc +// assembly-config.adoc [id='proc-manual-stop-pause-connector-{context}'] = Manually stopping or pausing Kafka Connect connectors diff --git a/documentation/modules/configuring/proc-resetting-connector-offsets.adoc b/documentation/modules/configuring/proc-resetting-connector-offsets.adoc new file mode 100644 index 00000000000..a813dc85e56 --- /dev/null +++ b/documentation/modules/configuring/proc-resetting-connector-offsets.adoc @@ -0,0 +1,87 @@ +// Module included in the following assemblies: +// +// assembly-config.adoc + +[id='proc-resetting-connector-offsets-{context}'] += Resetting connector offsets + +[role="_abstract"] +To reset connector offsets using `KafkaConnector` resources, annotate the resource to clear the offsets. + +In this procedure, we reset the offset position for a source connector named `my-source-connector`. + +NOTE: You can also alter connector offsets using the `DELETE /connectors/{connector}/offsets` request through the Kafka Connect Rest API. + +.Prerequisites + +* The Cluster Operator is running. + +.Procedure + +. Edit the `KafkaConnector` resource to stop the connector. ++ +.Example configuration to stop a connector +[source,yaml,subs="+attributes"] +---- +apiVersion: {KafkaConnectorApiVersion} +kind: KafkaConnector +metadata: + name: my-source-connector + labels: + strimzi.io/cluster: my-connect-cluster +spec: + # ... + state: stopped # <1> + # ... +---- +<1> Changes the state of the connector to `stopped`. The default state for the connector when this property is not set is `running`. + +. Run the command to reset the offset position by annotating the `KafkaConnector` resource: ++ +[source,shell] +---- +kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=reset -n +---- ++ +The annotation remains until either the reset operation succeeds, or it is manually removed from the resource. + +. Inspect the contents of the config map to verify the reset of the offset position: ++ +[source,shell] +---- +kubectl describe configmap my-connector-offsets -n +---- ++ +Strimzi puts the offsets into the `offsets.json` property. +It doesn't overwrite any other properties when updating an existing config map. +After resetting, the `offsets.json` property is empty. ++ +.Example source connector offset list +[source,yaml,subs="+attributes"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + # ... +data: + offsets.json: |- + { + "offsets" : [] + } +---- + +. Restart the connector by changing the state to `running`. ++ +.Example configuration to start a connector +[source,yaml,subs="+attributes"] +---- +apiVersion: {KafkaConnectorApiVersion} +kind: KafkaConnector +metadata: + name: my-source-connector + labels: + strimzi.io/cluster: my-connect-cluster +spec: + state: running + # ... +---- \ No newline at end of file diff --git a/documentation/modules/oauth/proc-oauth-authorization-keycloak-example.adoc b/documentation/modules/oauth/proc-oauth-authorization-keycloak-example.adoc index 45412d3d07d..2c632240db8 100644 --- a/documentation/modules/oauth/proc-oauth-authorization-keycloak-example.adoc +++ b/documentation/modules/oauth/proc-oauth-authorization-keycloak-example.adoc @@ -119,7 +119,7 @@ An affirmative policy means that at least one policy must be satisfied for a cli Groups:: `Groups` organize users and set permissions. Groups can be linked to an LDAP identity provider and used to compartmentalize users, such as by region or department. Users can be added to groups through a custom LDAP interface to manage permissions for Kafka resources. -For more information on using KeyCloak's LDAP provider, see the the guide to {keycloak-admin-guide}. +For more information on using KeyCloak's LDAP provider, see the guide to {keycloak-admin-guide}. Users:: `Users` define individual users. In this example, `alice` (in the `ClusterManager` group) and `bob` (in `ClusterManager-my-cluster`) are created. diff --git a/documentation/modules/upgrading/con-upgrade-status.adoc b/documentation/modules/upgrading/con-upgrade-status.adoc index 5a5e7a7d0e8..c1a1e608a07 100644 --- a/documentation/modules/upgrading/con-upgrade-status.adoc +++ b/documentation/modules/upgrading/con-upgrade-status.adoc @@ -12,7 +12,7 @@ The status provides information on the Strimzi and Kafka versions being used. To ensure that you have the correct versions after completing an upgrade, verify the `kafkaVersion` and `operatorLastSuccessfulVersion` values in the Kafka status. * `operatorLastSuccessfulVersion` is the version of the Strimzi operator that last performed a successful reconciliation. -* `kafkaVersion` is the the version of Kafka being used by the Kafka cluster. +* `kafkaVersion` is the version of Kafka being used by the Kafka cluster. * `kafkaMetadataVersion` is the metadata version used by KRaft-based Kafka clusters You can use these values to check an upgrade of Strimzi or Kafka has completed. From 2ce74db7a9c8269794f239b21f346bb8b5f9f0f0 Mon Sep 17 00:00:00 2001 From: prmellor Date: Thu, 3 Oct 2024 10:01:58 +0100 Subject: [PATCH 2/3] docs(review): edits to doc from comments by JS --01 Signed-off-by: prmellor --- .../proc-altering-connector-offsets.adoc | 44 +++---------------- .../proc-listing-connector-offsets.adoc | 5 +-- .../proc-resetting-connector-offsets.adoc | 15 ++----- 3 files changed, 13 insertions(+), 51 deletions(-) diff --git a/documentation/modules/configuring/proc-altering-connector-offsets.adoc b/documentation/modules/configuring/proc-altering-connector-offsets.adoc index c16683a1ecc..2fc9ce9bcf8 100644 --- a/documentation/modules/configuring/proc-altering-connector-offsets.adoc +++ b/documentation/modules/configuring/proc-altering-connector-offsets.adoc @@ -6,15 +6,14 @@ = Altering connector offsets [role="_abstract"] -To alter connector offsets using `KafkaConnector` resources, add the `alterOffsets` configuration. -This can be useful, for example, to skip a _poison_ record or replay a record. -The offset changes are specified in a config map. +To alter connector offsets using `KafkaConnector` resources, configure the resource to stop the connector and add `alterOffsets` configuration to specify the offset changes in a config map. You can reuse the same config map used to xref:proc-listing-connector-offsets-{context}[list offsets]. -The offset alteration is applied by annotating the `KafkaConnector` resource. -In this procedure, we alter the offset position for a source connector named `my-source-connector`. +After the connector is stopped and the configuration is in place, annotate the `KafkaConnector` resource to apply the offset alteration, then restart the connector. + +Altering connector offsets can be useful, for example, to skip a _poison_ record or replay a record. -NOTE: You can also alter connector offsets using the `PATCH /connectors/{connector}/offsets` request through the Kafka Connect Rest API. +In this procedure, we alter the offset position for a source connector named `my-source-connector`. .Prerequisites @@ -22,7 +21,7 @@ NOTE: You can also alter connector offsets using the `PATCH /connectors/{connect .Procedure -. Edit the `KafkaConnector` resource to stop the connector and include the `listOffsets` configuration. +. Edit the `KafkaConnector` resource to stop the connector and include the `alterOffsets` configuration. + .Example configuration to stop a connector and alter offsets [source,yaml,subs="+attributes"] @@ -79,36 +78,7 @@ kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets + The annotation remains until either the update operation succeeds, or it is manually removed from the resource. -. Inspect the contents of the config map to verify the update to the offset position: -+ -[source,shell] ----- -kubectl describe configmap my-connector-offsets -n ----- -+ -Strimzi puts the offsets into the `offsets.json` property. -It doesn't overwrite any other properties when updating an existing config map. -+ -.Example source connector offset list -[source,yaml,subs="+attributes"] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - # ... -data: - offsets.json: |- - { - "offsets" : [ { - "partition" : { - "filename" : "/data/myfile.txt" - }, - "offset" : { - "position" : 15000 - } - } ] - } ----- +. Check the changes by using the procedure to xref:proc-listing-connector-offsets-{context}[list connector offsets]. . Restart the connector by changing the state to `running`. + diff --git a/documentation/modules/configuring/proc-listing-connector-offsets.adoc b/documentation/modules/configuring/proc-listing-connector-offsets.adoc index 478fcc6dfcd..52c0f4c3608 100644 --- a/documentation/modules/configuring/proc-listing-connector-offsets.adoc +++ b/documentation/modules/configuring/proc-listing-connector-offsets.adoc @@ -9,15 +9,14 @@ To track connector offsets using `KafkaConnector` resources, add the `listOffsets` configuration. The offsets, which keep track of the flow of data, are written to a config map specified in the configuration. If the config map does not exist, Strimzi creates it. -The list is written to the config map by annotating the `KafkaConnector` resource. + +After the configuration is in place, annotate the `KafkaConnector` resource to write the list to the config map. Sink connectors use Kafka's standard consumer offset mechanism, while source connectors store offsets in a custom format within a Kafka topic. * For sink connectors, the list shows Kafka topic partitions and the last committed offset for each partition. * For source connectors, the list shows the source system’s partition and the last offset processed. -NOTE: You can also list connector offsets using the `GET /connectors/{connector}/offsets` request through the Kafka Connect Rest API. - .Prerequisites * The Cluster Operator is running. diff --git a/documentation/modules/configuring/proc-resetting-connector-offsets.adoc b/documentation/modules/configuring/proc-resetting-connector-offsets.adoc index a813dc85e56..b54a45a02aa 100644 --- a/documentation/modules/configuring/proc-resetting-connector-offsets.adoc +++ b/documentation/modules/configuring/proc-resetting-connector-offsets.adoc @@ -6,11 +6,11 @@ = Resetting connector offsets [role="_abstract"] -To reset connector offsets using `KafkaConnector` resources, annotate the resource to clear the offsets. +To reset connector offsets using `KafkaConnector` resources, configure the resource to stop the connector. -In this procedure, we reset the offset position for a source connector named `my-source-connector`. +After the connector is stopped, annotate the `KafkaConnector` resource to clear the offsets, then restart the connector. -NOTE: You can also alter connector offsets using the `DELETE /connectors/{connector}/offsets` request through the Kafka Connect Rest API. +In this procedure, we reset the offset position for a source connector named `my-source-connector`. .Prerequisites @@ -45,15 +45,8 @@ kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets + The annotation remains until either the reset operation succeeds, or it is manually removed from the resource. -. Inspect the contents of the config map to verify the reset of the offset position: -+ -[source,shell] ----- -kubectl describe configmap my-connector-offsets -n ----- +. Check the changes by using the procedure to xref:proc-listing-connector-offsets-{context}[list connector offsets]. + -Strimzi puts the offsets into the `offsets.json` property. -It doesn't overwrite any other properties when updating an existing config map. After resetting, the `offsets.json` property is empty. + .Example source connector offset list From 70e8f52dad1e7cbb5d6c80f9f6de3246ca8c5563 Mon Sep 17 00:00:00 2001 From: prmellor Date: Fri, 11 Oct 2024 09:47:42 +0100 Subject: [PATCH 3/3] docs(review): edits to doc from comments by KS Signed-off-by: prmellor --- .../con-config-mirrormaker2-connectors.adoc | 65 ++++++++++++++++++- .../proc-altering-connector-offsets.adoc | 9 +-- .../proc-listing-connector-offsets.adoc | 10 +-- .../proc-resetting-connector-offsets.adoc | 2 +- 4 files changed, 74 insertions(+), 12 deletions(-) diff --git a/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc b/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc index 18717953c8a..32f1afb7198 100644 --- a/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc +++ b/documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc @@ -247,11 +247,12 @@ Mismatched or missing settings cause issues with data replication or offset sync == Listing the offsets of MirrorMaker 2 connectors To list the offset positions of the internal MirrorMaker 2 connectors, use the same configuration that's used to manage Kafka Connect connectors. +For more information on setting up the configuration and listing offsets, see xref:proc-listing-connector-offsets-{context}[]. In this example, the `sourceConnector` configuration is updated to return the connector offset position. The offset information is written to a specified config map. -.Example configuration for connector producers and consumers +.Example configuration for MirrorMaker 2 connector [source,yaml,subs="+quotes,attributes"] ---- apiVersion: {KafkaMirrorMaker2ApiVersion} @@ -261,6 +262,11 @@ metadata: spec: version: {DefaultKafkaVersion} # ... + clusters: + - alias: "my-cluster-source" + bootstrapServers: my-cluster-source-kafka-bootstrap:9092 + - alias: "my-cluster-target" + bootstrapServers: my-cluster-target-kafka-bootstrap:9092 mirrors: - sourceCluster: "my-cluster-source" targetCluster: "my-cluster-target" @@ -271,6 +277,61 @@ spec: # ... ---- -For more information on setting up the configuration, see xref:proc-listing-connector-offsets-{context}[]. +You must apply the following annotations to the `KafkaMirrorMaker2` resource be able to manage connector offsets: + +* `strimzi.io/connector-offsets` +* `strimzi.io/mirrormaker-connector` + +The `strimzi.io/mirrormaker-connector` annotation must be set to the name of the connector. +These annotations remain until the operation succeeds or they are manually removed from the resource. + +MirrorMaker 2 connectors are named using the aliases of the source and target clusters, followed by the connector type: `->.`. + +In the following example, the annotations are applied for a connector named `my-cluster-source->my-cluster-target.MirrorSourceConnector`. + +.Example application of annotations for connector +[source,shell] +---- +kubectl annotate kafkamirrormaker2 my-mirror-maker-2 strimzi.io/connector-offsets=list strimzi.io/mirrormaker-connector="my-cluster-source->my-cluster-target.MirrorSourceConnector" -n kafka +---- + +The offsets are listed in the specified config map. +Strimzi puts the offset information into a `.json` property named after the connector. +This does not overwrite any other properties when updating an existing config map. + +.Example source connector offset list +[source,yaml,subs="+attributes"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + # ... + ownerReferences: # <1> + - apiVersion: {KafkaMirrorMaker2ApiVersion} + blockOwnerDeletion: false + controller: false + kind: KafkaMirrorMaker2 + name: my-mirror-maker2 + uid: 637e3be7-bd96-43ab-abde-c55b4c4550e0 +data: + my-cluster-source--my-cluster-target.MirrorSourceConnector.json: |- # <2> + { + "offsets": [ + { + "partition": { + "cluster": "east-kafka", + "partition": 0, + "topic": "mirrormaker2-cluster-configs" + }, + "offset": { + "offset": 0 + } + } + ] + } +---- +<1> The owner reference pointing to the `KafkaMirrorMaker2` resource. +To provide a custom owner reference, create the config map in advance and set the owner reference. +<2> The `.json` property uses the connector name. Since `->` characters are not allowed in config map keys, `->` is changed to `--` in the connector name. NOTE: It is possible to use configuration to xref:proc-altering-connector-offsets-{context}[alter] or xref:proc-resetting-connector-offsets-{context}[reset] connector offsets, though this is rarely necessary. \ No newline at end of file diff --git a/documentation/modules/configuring/proc-altering-connector-offsets.adoc b/documentation/modules/configuring/proc-altering-connector-offsets.adoc index 2fc9ce9bcf8..b2494cd8dd5 100644 --- a/documentation/modules/configuring/proc-altering-connector-offsets.adoc +++ b/documentation/modules/configuring/proc-altering-connector-offsets.adoc @@ -55,19 +55,20 @@ kind: ConfigMap metadata: # ... data: - offsets.json: |- + offsets.json: |- # <1> { "offsets" : [ { "partition" : { "filename" : "/data/myfile.txt" }, "offset" : { - "position" : 15000 # <1> + "position" : 15000 # <2> } } ] } ---- -<1> The updated offset position in the source partition. +<1> Edits must be made within the `offsets.json` property. +<2> The updated offset position in the source partition. . Run the command to update the offset position by annotating the `KafkaConnector` resource: + @@ -76,7 +77,7 @@ data: kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=alter -n ---- + -The annotation remains until either the update operation succeeds, or it is manually removed from the resource. +The annotation remains until either the update operation succeeds or it is manually removed from the resource. . Check the changes by using the procedure to xref:proc-listing-connector-offsets-{context}[list connector offsets]. diff --git a/documentation/modules/configuring/proc-listing-connector-offsets.adoc b/documentation/modules/configuring/proc-listing-connector-offsets.adoc index 52c0f4c3608..5b6f292a8c4 100644 --- a/documentation/modules/configuring/proc-listing-connector-offsets.adoc +++ b/documentation/modules/configuring/proc-listing-connector-offsets.adoc @@ -50,7 +50,7 @@ spec: kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=list -n ---- + -The annotation remains until either the list operation succeeds, or it is manually removed from the resource. +The annotation remains until either the list operation succeeds or it is manually removed from the resource. . After the `KafkaConnector` resource is updated, use the following command to check if the config map with the offsets was created: + @@ -66,8 +66,8 @@ kubectl get configmap my-connector-offsets -n kubectl describe configmap my-connector-offsets -n ---- + -Strimzi puts the offsets into the `offsets.json` property. -It doesn't overwrite any other properties when updating an existing config map. +Strimzi puts the offset information into the `offsets.json` property. +This does not overwrite any other properties when updating an existing config map. + -- .Example source connector offset list @@ -100,7 +100,7 @@ data: } ---- <1> The owner reference pointing to the `KafkaConnector` resource for the source connector. -To provide a custom owner reference, create the config map in advance and set the owner reference. +To provide a custom owner reference, create the config map in advance and set the owner reference. <2> The source partition, represented by the filename `/data/myfile.txt` in this example for a file-based connector. <3> The last processed offset position in the source partition. -- @@ -129,7 +129,7 @@ data: { "partition": { "kafka_topic": "my-topic", # <2> - "kafka_partition": 2 # <3> + "kafka_partition": 2 # <3> }, "offset": { "kafka_offset": 4 # <4> diff --git a/documentation/modules/configuring/proc-resetting-connector-offsets.adoc b/documentation/modules/configuring/proc-resetting-connector-offsets.adoc index b54a45a02aa..84ae52f4e57 100644 --- a/documentation/modules/configuring/proc-resetting-connector-offsets.adoc +++ b/documentation/modules/configuring/proc-resetting-connector-offsets.adoc @@ -43,7 +43,7 @@ spec: kubectl annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=reset -n ---- + -The annotation remains until either the reset operation succeeds, or it is manually removed from the resource. +The annotation remains until either the reset operation succeeds or it is manually removed from the resource. . Check the changes by using the procedure to xref:proc-listing-connector-offsets-{context}[list connector offsets]. +