[CELEBORN-1801] Remove out-of-dated flink 1.14 and 1.15

### What changes were proposed in this pull request? Remove out-of-dated flink 1.14 and 1.15. For more information, please see the discussion thread: https://lists.apache.org/thread/njho00zmkjx5qspcrbrkogy8s4zzmwv9 ### Why are the changes needed? Reduce maintenance burden. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? Changes can be covered by existing tests. Closes #3029 from codenohup/remove-flink14and15. Authored-by: codenohup <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
apache · Dec 30, 2024 · a572380 · a572380
1 parent d0d8edf
commit a572380
Show file tree

Hide file tree

Showing 57 changed files with 25 additions and 5,798 deletions.
diff --git a/.github/workflows/deps.yml b/.github/workflows/deps.yml
@@ -48,8 +48,6 @@ jobs:
           - 'spark-3.3'
           - 'spark-3.4'
           - 'spark-3.5'
-          - 'flink-1.14'
-          - 'flink-1.15'
           - 'flink-1.16'
           - 'flink-1.17'
           - 'flink-1.18'
@@ -82,8 +80,6 @@ jobs:
           - 'spark-3.3'
           - 'spark-3.4'
           - 'spark-3.5'
-          - 'flink-1.14'
-          - 'flink-1.15'
           - 'flink-1.16'
           - 'flink-1.17'
           - 'flink-1.18'

diff --git a/.github/workflows/license.yml b/.github/workflows/license.yml
@@ -43,8 +43,6 @@ jobs:
           cache: 'maven'
           check-latest: false
       - run: |
-          build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.14
-          build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.15
           build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.16
           build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.17
           build/mvn org.apache.rat:apache-rat-plugin:check -Pgoogle-mirror,flink-1.18

diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml
@@ -159,8 +159,6 @@ jobs:
           - 8
           - 11
         flink:
-          - '1.14'
-          - '1.15'
           - '1.16'
           - '1.17'
           - '1.18'

diff --git a/.github/workflows/sbt.yml b/.github/workflows/sbt.yml
@@ -205,8 +205,6 @@ jobs:
           - 8
           - 11
         flink:
-          - '1.14'
-          - '1.15'
           - '1.16'
           - '1.17'
           - '1.18'

diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml
@@ -47,8 +47,6 @@ jobs:
           cache: 'maven'
           check-latest: false
       - run: |
-          build/mvn spotless:check -Pgoogle-mirror,flink-1.14
-          build/mvn spotless:check -Pgoogle-mirror,flink-1.15
           build/mvn spotless:check -Pgoogle-mirror,flink-1.16
           build/mvn spotless:check -Pgoogle-mirror,flink-1.17
           build/mvn spotless:check -Pgoogle-mirror,flink-1.18

diff --git a/README.md b/README.md
@@ -41,12 +41,12 @@ Celeborn Worker's slot count is decided by `total usable disk size / average shu
 Celeborn worker's slot count decreases when a partition is allocated and increments when a partition is freed.
 
 ## Build
-1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4/3.5, Flink 1.14/1.15/1.16/1.17/1.18/1.19/1.20 and Hadoop MapReduce 2/3.
+1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4/3.5, Flink 1.16/1.17/1.18/1.19/1.20 and Hadoop MapReduce 2/3.
 2. Celeborn tested under Scala 2.11/2.12/2.13 and Java 8/11/17 environment.
 
 Build Celeborn via `make-distribution.sh`:
 ```shell
-./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pflink-1.14/-Pflink-1.15/-Pflink-1.16/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pmr
+./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pflink-1.16/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pmr
 ```
 
 Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
@@ -62,8 +62,6 @@ Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
 | Spark 3.3  | &#x274C;          | &#10004;          | &#10004;           | &#10004;           | &#10004;          | &#10004;           | &#10004;           |
 | Spark 3.4  | &#x274C;          | &#10004;          | &#10004;           | &#10004;           | &#10004;          | &#10004;           | &#10004;           |
 | Spark 3.5  | &#x274C;          | &#10004;          | &#10004;           | &#10004;           | &#10004;          | &#10004;           | &#10004;           |
-| Flink 1.14 | &#x274C;          | &#10004;          | &#10004;           | &#x274C;           | &#x274C;          | &#x274C;           | &#x274C;           |
-| Flink 1.15 | &#x274C;          | &#10004;          | &#10004;           | &#x274C;           | &#x274C;          | &#x274C;           | &#x274C;           |
 | Flink 1.16 | &#x274C;          | &#10004;          | &#10004;           | &#x274C;           | &#x274C;          | &#x274C;           | &#x274C;           |
 | Flink 1.17 | &#x274C;          | &#10004;          | &#10004;           | &#x274C;           | &#x274C;          | &#x274C;           | &#x274C;           |
 | Flink 1.18 | &#x274C;          | &#10004;          | &#10004;           | &#x274C;           | &#x274C;          | &#x274C;           | &#x274C;           |
@@ -114,7 +112,7 @@ Celeborn server is compatible with all clients inside various engines.
 However, Celeborn clients must be consistent with the version of the specified engine.
 For example, if you are running Spark 2.4, you must compile Celeborn client with -Pspark-2.4;
 if you are running Spark 3.2, you must compile Celeborn client with -Pspark-3.2; 
-if you are running flink 1.14, you must compile Celeborn client with -Pflink-1.14.
+if you are running flink 1.16, you must compile Celeborn client with -Pflink-1.16.
 
 ## Usage
 Celeborn cluster composes of Master and Worker nodes, the Master supports both single and HA mode(Raft-based) deployments.
@@ -318,7 +316,7 @@ spark.executor.userClassPathFirst false
 Copy `$CELEBORN_HOME/flink/*.jar` to `$FLINK_HOME/lib/`.
 
 #### Flink Configuration
-Celeborn supports two Flink integration strategies: remote shuffle service (since Flink 1.14) and [hybrid shuffle](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/batch/batch_shuffle/#hybrid-shuffle) (since Flink 1.20).
+Celeborn supports two Flink integration strategies: remote shuffle service (since Flink 1.16) and [hybrid shuffle](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/batch/batch_shuffle/#hybrid-shuffle) (since Flink 1.20).
 
 To use Celeborn, you can choose one of them and add the following Flink configurations.
 

diff --git a/build/make-distribution.sh b/build/make-distribution.sh
@@ -342,8 +342,6 @@ if [ "$SBT_ENABLED" == "true" ]; then
     sbt_build_client -Pspark-2.4
     sbt_build_client -Pspark-3.4
     sbt_build_client -Pspark-3.5
-    sbt_build_client -Pflink-1.14
-    sbt_build_client -Pflink-1.15
     sbt_build_client -Pflink-1.16
     sbt_build_client -Pflink-1.17
     sbt_build_client -Pflink-1.18
@@ -378,8 +376,6 @@ else
     build_spark_client -Pspark-2.4
     build_spark_client -Pspark-3.4
     build_spark_client -Pspark-3.5
-    build_flink_client -Pflink-1.14
-    build_flink_client -Pflink-1.15
     build_flink_client -Pflink-1.16
     build_flink_client -Pflink-1.17
     build_flink_client -Pflink-1.18

diff --git a/build/release/release.sh b/build/release/release.sh
@@ -104,12 +104,6 @@ upload_nexus_staging() {
   echo "Deploying celeborn-client-spark-3-shaded_2.13"
   ${PROJECT_DIR}/build/sbt -Pspark-3.4 ++2.13 "clean;celeborn-client-spark-3-shaded/publishSigned"
 
-  echo "Deploying celeborn-client-flink-1.14-shaded_2.12"
-  ${PROJECT_DIR}/build/sbt -Pflink-1.14 "clean;celeborn-client-flink-1_14-shaded/publishSigned"
-
-  echo "Deploying celeborn-client-flink-1.15-shaded_2.12"
-  ${PROJECT_DIR}/build/sbt -Pflink-1.15 "clean;celeborn-client-flink-1_15-shaded/publishSigned"
-
   echo "Deploying celeborn-client-flink-1.16-shaded_2.12"
   ${PROJECT_DIR}/build/sbt -Pflink-1.16 "clean;celeborn-client-flink-1_16-shaded/publishSigned"
 

diff --git a/client-flink/flink-1.14-shaded/pom.xml b/client-flink/flink-1.14-shaded/pom.xml
-Original file line number
+Diff line change
@@ Expand Up / @@ -159,8 +159,6 @@ jobs: @@
               - 8
               - 11
             flink:
-              - '1.14'
-              - '1.15'
               - '1.16'
               - '1.17'
               - '1.18'
@@ Expand Down @@