Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink #6585

Merged
merged 6 commits into from
Apr 8, 2024
Merged

[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink #6585

merged 6 commits into from
Apr 8, 2024

Conversation

dailai
Copy link
Contributor

@dailai dailai commented Mar 27, 2024

Purpose of this pull request

  1. Support hadoop-ha and kerberos for paimon sink
  2. Fix the bug of date type in paimon sink. We should convert date type to int when we sink paimon ranther then timestamp. Refer: org.apache.paimon.data.BinaryWriter#createValueSetter. Otherwise, when we use flink to query the date data in paimon, the date data is not correct

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

hailin0
hailin0 reviewed Mar 27, 2024
View reviewed changes
docs/en/connector-v2/sink/Paimon.md
@@ -17,12 +17,13 @@ Sink connector for Apache Paimon. It can support cdc mode 、auto create table.
| warehouse | String | Yes | - | Paimon warehouse path |
| database | String | Yes | - | The database you want to access |
| table | String | Yes | - | The table you want to access |
| hdfs_site_path | String | No | - | |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be compatible with older versions

Copy link
Contributor Author

@dailai dailai Mar 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be compatible with older versions

Yes, the code is compatible with older versions. I've updated the docs.

@dailai dailai changed the title [Feature][Connector-V2] Support hadoop ha and kerberos for paimon sink [Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink Mar 27, 2024
@hailin0 hailin0 added this to the 2.3.5 milestone Mar 29, 2024
@dailai dailai requested a review from hailin0 April 1, 2024 05:31
@dailai dailai force-pushed the support-hadoop-ha-and-kerberos branch from 8cc8559 to 7025b05 Compare April 3, 2024 01:04
@dailai dailai requested a review from hailin0 April 3, 2024 08:22
@hailin0
Copy link
Member

hailin0 commented Apr 7, 2024

LGTM

cc @EricJoy2048

@Carl-Zhou-CN
Copy link
Member

link #6076

@Carl-Zhou-CN
Copy link
Member

@dailai hi,Since ci cannot test your test case, can you run the test case locally and provide screenshots

Carl-Zhou-CN
Carl-Zhou-CN reviewed Apr 7, 2024
View reviewed changes
...imon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/config/PaimonConfig.java
return argument;
}

@VisibleForTesting
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this annotation? It is not intended for testing purposes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Carl-Zhou-CN
This annotation is meaningful to prevent method permissions from being changed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, it's my oversight

@dailai
Copy link
Contributor Author

dailai commented Apr 8, 2024

@dailai hi,Since ci cannot test your test case, can you run the test case locally and provide screenshots

截屏2024-04-08 08 00 03
截屏2024-04-08 08 00 17

@dailai dailai requested a review from Carl-Zhou-CN April 8, 2024 00:02
Carl-Zhou-CN
Carl-Zhou-CN approved these changes Apr 8, 2024
View reviewed changes
Copy link
Member

@Carl-Zhou-CN Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Carl-Zhou-CN
Copy link
Member

cc @EricJoy2048

@EricJoy2048
Copy link
Member

LGTM, Thanks for your contribution.

@EricJoy2048 EricJoy2048 merged commit 20b62f3 into apache:dev Apr 8, 2024
9 checks passed
@dailai dailai deleted the support-hadoop-ha-and-kerberos branch April 10, 2024 06:56
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
@dailai
[Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink (
893c90b 
apache#6585)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Carl-Zhou-CN Carl-Zhou-CN Carl-Zhou-CN approved these changes

@EricJoy2048 EricJoy2048 EricJoy2048 approved these changes

@hailin0 hailin0 Awaiting requested review from hailin0

Assignees
No one assigned
Labels
approved improve paimon reviewed
Projects
None yet
Milestone
2.3.5
Development

Successfully merging this pull request may close these issues.
4 participants
@dailai @hailin0 @Carl-Zhou-CN @EricJoy2048