align sbt with maven build

apache · Dec 29, 2024 · 21832a7 · 21832a7
1 parent cee1ea7
commit 21832a7
Show file tree

Hide file tree

Showing 2 changed files with 24 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -378,6 +378,25 @@ Meanwhile, configure the following settings in YARN and MapReduce config.
 **Note**: `MRAppMasterWithCeleborn` supports setting `mapreduce.celeborn.master.endpoints` via environment variable `CELEBORN_MASTER_ENDPOINTS`.
 Meanwhile, `MRAppMasterWithCeleborn` disables `yarn.app.mapreduce.am.job.recovery.enable` and sets `mapreduce.job.reduce.slowstart.completedmaps` to 1 by default.
 
+
+### Deploy Tez client
+Copy `$CELEBORN_HOME/tez/*.jar` into `mapreduce.application.classpath` and `yarn.application.classpath`.
+Meanwhile, configure the following settings in hive-site config and append `org.apache.tez.dag.app.CelebornDagAppMaster` to your `tez.am.launch.cmd-opts` in the `tez-site.xml`  
+```properties
+tez.celeborn.master.endpoints <master-1-1>:9097
+```
+**Note**: `CelebornDagAppMaster` supports setting `tez.celeborn.master.endpoints` via environment variable `CELEBORN_MASTER_ENDPOINTS`.
+Meanwhile, `CelebornDagAppMaster` will override following configs in tez-site.xml.
+
+| config item                                              | override value |
+|----------------------------------------------------------|----------------|
+| tez.shuffle-vertex-manager.min-src-fraction              | 1.0f           |
+| tez.shuffle-vertex-manager.max-src-fraction              | 1.0f           |
+| tez.runtime.transfer.data-via-events.enabled             | false          |
+| tez.runtime.transfer.data-via-events.support.in-mem.file | false          |
+| tez.runtime.pipelined-shuffle.enabled                    | false          |
+| tez.am.node-unhealthy-reschedule-tasks                   | false          |
+
 ### Best Practice
 If you want to set up a production-ready Celeborn cluster, your cluster should have at least 3 masters and at least 4 workers.
 Masters and works can be deployed on the same node but should not deploy multiple masters or workers on the same node.

diff --git a/project/CelebornBuild.scala b/project/CelebornBuild.scala
@@ -1706,6 +1706,7 @@ object TezClientProjects {
               name.startsWith("RoaringBitmap-") ||
               name.startsWith("lz4-java-") ||
               name.startsWith("zstd-jni-") ||
+              name.startsWith("metrics-core-") ||
               name.startsWith("scala-library-"))
           }
         },
@@ -1715,7 +1716,10 @@ object TezClientProjects {
           ShadeRule.rename("com.google.common.**" -> "org.apache.celeborn.shaded.com.google.common.@1").inAll,
           ShadeRule.rename("io.netty.**" -> "org.apache.celeborn.shaded.io.netty.@1").inAll,
           ShadeRule.rename("org.apache.commons.**" -> "org.apache.celeborn.shaded.org.apache.commons.@1").inAll,
-          ShadeRule.rename("org.roaringbitmap.**" -> "org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
+          ShadeRule.rename("org.roaringbitmap.**" -> "org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll,
+          ShadeRule.rename("io.dropwizard.metrics.**" -> "org.apache.celeborn.shaded.io.dropwizard.metrics.@1").inAll,
+          ShadeRule.rename("com.codahale.metrics.**" -> "org.apache.celeborn.shaded.com.codahale.metrics.@1").inAll,
+          ShadeRule.rename("com.github.luben.**" -> "org.apache.celeborn.shaded.com.github.luben.@1").inAll,
         ),
 
         (assembly / assemblyMergeStrategy) := {