Skip to content

Commit

Permalink
[CELEBORN-1684] Fix ambiguous client jar expression of document
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
When users deploy using the release binary as outlined in the documentation, the instructions for copying the client JAR can be unclear.

### Why are the changes needed?
No

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
![image](https://github.com/user-attachments/assets/a4e7c415-8f0e-44bd-8d18-18462896e27c)

Closes apache#2877 from zaynt4606/md.

Authored-by: szt <[email protected]>
Signed-off-by: SteNicholas <[email protected]>
  • Loading branch information
zaynt4606 authored and SteNicholas committed Nov 5, 2024
1 parent b5201df commit ec67366
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 7 deletions.
8 changes: 4 additions & 4 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ WorkerRef: null
Celeborn release binary contains clients for Spark 2.x and Spark 3.x, copy the corresponding client jar into Spark's
`jars/` directory:
```shell
cp $CELEBORN_HOME/spark/<Celeborn Client Jar> $SPARK_HOME/jars/
cp $CELEBORN_HOME/spark/celeborn-client-spark-<spark.major.version>-shaded_<scala.binary.version>-<celeborn.version>.jar $SPARK_HOME/jars/
```
#### Start spark-shell
Set `spark.shuffle.manager` to Celeborn's ShuffleManager, and turn off `spark.shuffle.service.enabled`:
Expand Down Expand Up @@ -125,7 +125,7 @@ INFO [async-reply] Controller: CommitFiles for local-1690000152711-0 success wit
Celeborn release binary contains clients for Flink 1.14.x, Flink 1.15.x, Flink 1.17.x, Flink 1.18.x, Flink 1.19.x and Flink 1.20.x, copy the corresponding client jar into Flink's
`lib/` directory:
```shell
cp $CELEBORN_HOME/flink/<Celeborn Client Jar> $FLINK_HOME/lib/
cp $CELEBORN_HOME/flink/celeborn-client-flink-<flink.version>-shaded_<scala.binary.version>-<celeborn.version>.jar $FLINK_HOME/lib/
```
#### Add Celeborn configuration to Flink's conf
Set `shuffle-service-factory.class` to Celeborn's ShuffleServiceFactory in Flink configuration file:
Expand Down Expand Up @@ -181,8 +181,8 @@ INFO [async-reply] Controller: CommitFiles for local-1690000152711-0 success wit
### Copy Celeborn Client to MapReduce's classpath
1. Copy `$CELEBORN_HOME/mr/*.jar` into `mapreduce.application.classpath` and `yarn.application.classpath`.
```shell
cp $CELEBORN_HOME/mr/<Celeborn Client Jar> <mapreduce.application.classpath>
cp $CELEBORN_HOME/mr/<Celeborn Client Jar> <yarn.application.classpath>
cp $CELEBORN_HOME/mr/celeborn-client-mr-shaded_<scala.binary.version>-<celeborn.version>.jar <mapreduce.application.classpath>
cp $CELEBORN_HOME/mr/celeborn-client-mr-shaded_<scala.binary.version>-<celeborn.version>.jar <yarn.application.classpath>
```
2. Restart your yarn cluster.
### Add Celeborn configuration to MapReduce's conf
Expand Down
13 changes: 10 additions & 3 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,10 @@ WorkerRef: null
```

## Deploy Spark client
Copy `$CELEBORN_HOME/spark/*.jar` to `$SPARK_HOME/jars/`.
Celeborn release binary contains clients for Spark 2.x and Spark 3.x, copy the corresponding client jar into Spark's
`jars/` directory:

Copy `$CELEBORN_HOME/spark/celeborn-client-spark-<spark.major.version>-shaded_<scala.binary.version>-<celeborn.version>.jar` to `$SPARK_HOME/jars/`.

### Spark Configuration
To use Celeborn, the following spark configurations should be added.
Expand Down Expand Up @@ -209,7 +212,10 @@ spark.executor.userClassPathFirst false

**Important: Only Flink batch jobs are supported for now.**

Copy `$CELEBORN_HOME/flink/*.jar` to `$FLINK_HOME/lib/`.
Celeborn release binary contains clients for Flink 1.14.x, Flink 1.15.x, Flink 1.17.x, Flink 1.18.x, Flink 1.19.x and Flink 1.20.x, copy the corresponding client jar into Flink's
`lib/` directory:

Copy `$CELEBORN_HOME/flink/celeborn-client-flink-<flink.version>-shaded_<scala.binary.version>-<celeborn.version>.jar` to `$FLINK_HOME/lib/`.

### Flink Configuration
Celeborn supports two Flink integration strategies: remote shuffle service (since Flink 1.14) and [hybrid shuffle](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/batch/batch_shuffle/#hybrid-shuffle) (since Flink 1.20).
Expand Down Expand Up @@ -259,7 +265,8 @@ celeborn.rpc.dispatcher.numThreads: 32
**Note**: The config option `execution.batch-shuffle-mode` should configure as `ALL_EXCHANGES_HYBRID_FULL`.

## Deploy MapReduce client
Copy `$CELEBORN_HOME/mr/*.jar` into `mapreduce.application.classpath` and `yarn.application.classpath`.
Copy `$CELEBORN_HOME/mr/celeborn-client-mr-shaded_<scala.binary.version>-<celeborn.version>.jar` into `mapreduce.application.classpath` and `yarn.application.classpath`.

Meanwhile, configure the following settings in YARN and MapReduce config.
```bash
-Dyarn.app.mapreduce.am.job.recovery.enable=false
Expand Down

0 comments on commit ec67366

Please sign in to comment.