[feature][Connector-V2][cdc] Support flink running cdc job #4918

Carl-Zhou-CN · 2023-06-14T01:04:51Z

Purpose of this pull request

close #4605

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

hailin0 · 2023-06-14T04:06:51Z

good

TyrantLucifer · 2023-06-15T05:57:12Z

good pr

EricJoy2048 · 2023-06-15T11:52:34Z

niubility!

hailin0 · 2023-07-09T08:00:23Z

waitting for ci check

Carl-Zhou-CN · 2023-07-09T08:21:27Z

@hailin0 Please help to start the client,the mysql cdc driver is abnormal

Carl-Zhou-CN · 2023-07-09T10:27:23Z

Appears to be a zeta startup exception

hailin0 · 2023-07-10T02:02:12Z

@EricJoy2048

Carl-Zhou-CN · 2023-07-12T04:36:46Z

@evan766 Can you share your json configuration?

Carl-Zhou-CN · 2023-07-12T06:51:28Z

Whether this was removed when merged？@evan766

Carl-Zhou-CN · 2023-07-12T08:36:10Z

@evan766 Can I add my wechat zy330600

hailin0 · 2023-07-25T13:41:34Z

Please merge dev branch into this pr

@Carl-Zhou-CN

Carl-Zhou-CN · 2023-07-26T07:21:43Z

@hailin0 ok

hailin0 · 2023-07-27T02:21:38Z

...erver-e2e/src/test/java/org/apache/seatunnel/e2e/connector/cdc/sqlserver/SqlServerCDCIT.java

@@ -65,7 +65,7 @@
 @Slf4j
 @DisabledOnContainer(
        value = {},
-        type = {EngineType.SPARK, EngineType.FLINK},
+        type = {EngineType.SPARK},


update testcase

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mongodb-e2e/src/test/java/mongodb/MongodbCDCIT.java#L68
https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java#L58

hailin0 · 2023-07-28T09:42:36Z

check ci error

Carl-Zhou-CN · 2023-08-09T10:23:19Z

Temporarily blocks mongo cdc effects #5251

Carl-Zhou-CN · 2023-08-14T06:44:05Z

In his own warehouse he was successful

Carl-Zhou-CN · 2023-08-25T01:09:54Z

@ic4y @hailin0 @EricJoy2048 @Hisoka-X @TyrantLucifer Please help review, thank you

hailin0 · 2023-09-12T05:36:15Z

LGTM

TyrantLucifer

LGTM, thank you for your contribution!

Hisoka-X · 2023-09-27T10:05:10Z

...a/org/apache/seatunnel/core/starter/flink/execution/FlinkAbstractPluginExecuteProcessor.java

+            if (pluginConfig.hasPath(SOURCE_TABLE_NAME)) {
+                String sourceTable = pluginConfig.getString(SOURCE_TABLE_NAME);
+                flinkRuntimeEnvironment.registerResultTable(
+                        pluginConfig,
+                        dataStream,
+                        resultTable,
+                        isAppendMap.getOrDefault(sourceTable, true));
+                registerAppendStream(pluginConfig);
+                return;
+            }
+            flinkRuntimeEnvironment.registerResultTable(
+                    pluginConfig,
+                    dataStream,
+                    resultTable,
+                    isAppendMap.getOrDefault(resultTable, true));


~~Hi @Carl-Zhou-CN , could you explain for me why we should register table as source table name when both have RESULT_TABLE_NAME and SOURCE_TABLE_NAME? Thanks.~~

I got it. Please ignore it.

* [feature][Connector-V2][cdc] Support flink running cdc job * [feature][Connector-V2][cdc] Support flink running cdc job * [feature][Connector-V2][cdc] mongo e2e delete --------- Co-authored-by: zhouyao <[email protected]>

liugddx closed this Jul 18, 2023

liugddx reopened this Jul 18, 2023

hailin0 closed this Jul 27, 2023

hailin0 reopened this Jul 27, 2023

hailin0 reviewed Jul 27, 2023

View reviewed changes

hailin0 added cdc flink labels Jul 27, 2023

EricJoy2048 added this to the 2.3.3 milestone Aug 4, 2023

ic4y removed this from the 2.3.3 milestone Aug 6, 2023

zhouyao added 3 commits August 14, 2023 16:11

[feature][Connector-V2][cdc] Support flink running cdc job

167c9dd

[feature][Connector-V2][cdc] Support flink running cdc job

05a2f6a

[feature][Connector-V2][cdc] mongo e2e delete

f621076

Carl-Zhou-CN force-pushed the flink-cdc branch from 08049cf to f621076 Compare August 14, 2023 08:16

Merge branch 'apache:dev' into flink-cdc

0339539

hailin0 added this to the 2.3.4 milestone Aug 23, 2023

hailin0 approved these changes Sep 12, 2023

View reviewed changes

github-actions bot added approved reviewed labels Sep 12, 2023

TyrantLucifer approved these changes Sep 12, 2023

View reviewed changes

TyrantLucifer merged commit 5e37883 into apache:dev Sep 12, 2023
54 checks passed

hailin0 mentioned this pull request Sep 12, 2023

[Feature][CDC] Support MongoDB CDC running on flink #5465

Closed

3 tasks

Hisoka-X reviewed Sep 27, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature][Connector-V2][cdc] Support flink running cdc job #4918

[feature][Connector-V2][cdc] Support flink running cdc job #4918

Carl-Zhou-CN commented Jun 14, 2023 •

edited by hailin0

Loading

hailin0 commented Jun 14, 2023

TyrantLucifer commented Jun 15, 2023

EricJoy2048 commented Jun 15, 2023

hailin0 commented Jul 9, 2023

Carl-Zhou-CN commented Jul 9, 2023

Carl-Zhou-CN commented Jul 9, 2023

hailin0 commented Jul 10, 2023

Carl-Zhou-CN commented Jul 12, 2023

Carl-Zhou-CN commented Jul 12, 2023 •

edited

Loading

Carl-Zhou-CN commented Jul 12, 2023

hailin0 commented Jul 25, 2023 •

edited

Loading

Carl-Zhou-CN commented Jul 26, 2023

hailin0 Jul 27, 2023

hailin0 commented Jul 28, 2023

Carl-Zhou-CN commented Aug 9, 2023

Carl-Zhou-CN commented Aug 14, 2023

Carl-Zhou-CN commented Aug 25, 2023

hailin0 commented Sep 12, 2023

TyrantLucifer left a comment

Hisoka-X Sep 27, 2023 •

edited

Loading

[feature][Connector-V2][cdc] Support flink running cdc job #4918

[feature][Connector-V2][cdc] Support flink running cdc job #4918

Conversation

Carl-Zhou-CN commented Jun 14, 2023 • edited by hailin0 Loading

Purpose of this pull request

Check list

hailin0 commented Jun 14, 2023

TyrantLucifer commented Jun 15, 2023

EricJoy2048 commented Jun 15, 2023

hailin0 commented Jul 9, 2023

Carl-Zhou-CN commented Jul 9, 2023

Carl-Zhou-CN commented Jul 9, 2023

hailin0 commented Jul 10, 2023

Carl-Zhou-CN commented Jul 12, 2023

Carl-Zhou-CN commented Jul 12, 2023 • edited Loading

Carl-Zhou-CN commented Jul 12, 2023

hailin0 commented Jul 25, 2023 • edited Loading

Carl-Zhou-CN commented Jul 26, 2023

hailin0 Jul 27, 2023

Choose a reason for hiding this comment

hailin0 commented Jul 28, 2023

Carl-Zhou-CN commented Aug 9, 2023

Carl-Zhou-CN commented Aug 14, 2023

Carl-Zhou-CN commented Aug 25, 2023

hailin0 commented Sep 12, 2023

TyrantLucifer left a comment

Choose a reason for hiding this comment

Hisoka-X Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

Carl-Zhou-CN commented Jun 14, 2023 •

edited by hailin0

Loading

Carl-Zhou-CN commented Jul 12, 2023 •

edited

Loading

hailin0 commented Jul 25, 2023 •

edited

Loading

Hisoka-X Sep 27, 2023 •

edited

Loading