Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CELEBORN-1737] Support build tez client package #3028

Closed
wants to merge 7 commits into from

Conversation

GH-Gloway
Copy link
Contributor

@GH-Gloway GH-Gloway commented Dec 24, 2024

What changes were proposed in this pull request?

Add Tez packaging script.

Why are the changes needed?

To support build tez client.

Does this PR introduce any user-facing change?

Yes, enable Celeborn with tez support.

How was this patch tested?

Cluster test.

Copy link

codecov bot commented Dec 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 32.68%. Comparing base (2eb4c23) to head (a470611).
Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3028      +/-   ##
==========================================
- Coverage   32.99%   32.68%   -0.30%     
==========================================
  Files         331      336       +5     
  Lines       19842    20032     +190     
  Branches     1787     1792       +5     
==========================================
+ Hits         6545     6546       +1     
- Misses      12933    13123     +190     
+ Partials      364      363       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@FMX FMX changed the title [CELEBORN-1737][BUILD]Include Tez client into binary release tarball [CELEBORN-1737] Support build tez client package Dec 25, 2024
@turboFei
Copy link
Member

turboFei commented Dec 29, 2024

do you plan to have 2 modules likes other clients?

client-tez/tez
client-tez/tez-shaded

And could you enable the tez dependency check?
https://github.com/apache/celeborn/blob/main/dev/dependencies.sh
https://github.com/apache/celeborn/blob/main/.github/workflows/deps.yml

Also, it is better to provide the checking result for the shaded jar.

jar -tf $SHADED_JAR | grep -v celeborn | grep '\.class$'

@FMX
Copy link
Contributor

FMX commented Dec 29, 2024

do you plan to have 2 modules likes other clients?

client-tez/tez
client-tez/tez-shaded

And could you enable the tez dependency check? https://github.com/apache/celeborn/blob/main/dev/dependencies.sh https://github.com/apache/celeborn/blob/main/.github/workflows/deps.yml

Also, it is better to provide the checking result for the shaded jar.

jar -tf $SHADED_JAR | grep -v celeborn | grep '\.class$'

I've updated this PR.
The dep check for SBT will come up with a new PR.


"${BUILD_COMMAND[@]}"

## flink spark client jars
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: seems typo

<module>tests/tez-it</module>
</modules>
<properties>
<tez.version>0.10.3</tez.version>
<tez.version>0.10.2</tez.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by design?

Copy link
Contributor

@FMX FMX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I've tested this PR on a cluster. Merged into main(v0.6.0).

@FMX FMX closed this in d0d8edf Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants