-
Notifications
You must be signed in to change notification settings - Fork 763
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* update ubuntu to 20.04 Signed-off-by: minmingzhu <[email protected]> * update CI actions and reduce Linear dataset Signed-off-by: minmingzhu <[email protected]> * Update build_and_run_all_spark_3.1_hadoop_3.2.sh * update spark.conf Signed-off-by: minmingzhu <[email protected]> * debug CI Signed-off-by: minmingzhu <[email protected]> * debug CI Signed-off-by: minmingzhu <[email protected]> * split benchmark list Signed-off-by: minmingzhu <[email protected]> * update split benchmark list Signed-off-by: minmingzhu <[email protected]> * update setup-cluster.sh Signed-off-by: minmingzhu <[email protected]> * Move the files required by CI to the actions directory Signed-off-by: minmingzhu <[email protected]> * add benchmarks.lst Signed-off-by: minmingzhu <[email protected]> --------- Signed-off-by: minmingzhu <[email protected]>
- Loading branch information
1 parent
62de01b
commit efc86df
Showing
18 changed files
with
445 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
.github/workflows/hibench_ci_spark2.4_hadoop_2.7_part_2.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
name: HiBench CI SPARK_2.4_HADOOP_2.7 | ||
|
||
on: [push, pull_request] | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-20.04 | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up JDK 1.8 | ||
uses: actions/setup-java@v1 | ||
with: | ||
java-version: 1.8 | ||
- name: Restore cached dependencies | ||
uses: actions/cache@v3 | ||
with: | ||
path: | | ||
# /var/cache/apt/archives/*.deb | ||
~/.m2/repository | ||
~/opt | ||
key: ${{ runner.os }}_spark_2.4_hadoop_2.7 | ||
restore-keys: | | ||
${{ runner.os }}- | ||
- name: Cluster spark_2.4_hadoop_2.7_hive_0.14 | ||
run: | | ||
${{github.workspace}}/actions/build_and_run_all_spark_2.4_hadoop_2.7_hive_0.14_part_2.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
.github/workflows/hibench_ci_spark3.0_hadoop_3.2_part_2.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
name: HiBench CI SPARK_3.0_HADOOP_3.2 | ||
|
||
on: [push, pull_request] | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-20.04 | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up JDK 1.8 | ||
uses: actions/setup-java@v1 | ||
with: | ||
java-version: 1.8 | ||
- name: Restore cached dependencies | ||
uses: actions/cache@v3 | ||
with: | ||
path: | | ||
# /var/cache/apt/archives/*.deb | ||
~/.m2/repository | ||
~/opt | ||
key: ${{ runner.os }}_spark_3.0_hadoop_3.2 | ||
restore-keys: | | ||
${{ runner.os }}- | ||
- name: Cluster spark_3.0_hadoop_3.2_hive_3.0 | ||
run: | | ||
${{github.workspace}}/actions/build_and_run_all_spark_3.0_hadoop_3.2_part_2.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
.github/workflows/hibench_ci_spark3.1_hadoop_3.2_part_2.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
name: HiBench CI SPARK_3.1_HADOOP_3.2 | ||
|
||
on: [push, pull_request] | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-20.04 | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Set up JDK 1.8 | ||
uses: actions/setup-java@v1 | ||
with: | ||
java-version: 1.8 | ||
- name: Restore cached dependencies | ||
uses: actions/cache@v3 | ||
with: | ||
path: | | ||
# /var/cache/apt/archives/*.deb | ||
~/.m2/repository | ||
~/opt | ||
key: ${{ runner.os }}_spark_3.1_hadoop_3.2 | ||
restore-keys: | | ||
${{ runner.os }}- | ||
- name: Cluster spark_3.1_hadoop_3.2_hive_3.0 | ||
run: | | ||
${{github.workspace}}/actions/build_and_run_all_spark_3.1_hadoop_3.2_part_2.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
micro.sleep | ||
micro.sort | ||
micro.terasort | ||
micro.wordcount | ||
micro.repartition | ||
micro.dfsioe | ||
|
||
sql.aggregation | ||
sql.join | ||
sql.scan | ||
|
||
websearch.nutchindexing | ||
websearch.pagerank |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
ml.bayes | ||
ml.kmeans | ||
ml.lr | ||
ml.als | ||
ml.pca | ||
ml.gbt | ||
ml.rf | ||
ml.svd | ||
ml.linear | ||
ml.lda | ||
ml.svm | ||
ml.gmm | ||
ml.correlation | ||
ml.summarizer | ||
|
||
graph.nweight |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
31 changes: 31 additions & 0 deletions
31
actions/build_and_run_all_spark_2.4_hadoop_2.7_hive_0.14_part_2.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
#!/usr/bin/env bash | ||
|
||
# exit when any command fails | ||
set -e | ||
|
||
export HADOOP_VERSION=2.7.7 | ||
export SPARK_VERSION=2.4.0 | ||
export SPARK_BIN_VERSION=spark2.4 | ||
export SPARK_HADOOP_VERSION=hadoop2.7 | ||
export HIVE_VERSION=0.14.0 | ||
|
||
# keep track of the last executed command | ||
trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG | ||
# echo an error message before exiting | ||
trap 'echo "\"${last_command}\" command filed with exit code $?."' EXIT | ||
|
||
# mvn build | ||
mvn clean package -q -Dmaven.javadoc.skip=true -Dspark=2.4 -Dscala=2.11 -Dhive=0.14 -Dhadoop=2.7 | ||
|
||
# Setup cluster contain of hadoop and spark | ||
source $GITHUB_WORKSPACE/actions/test-cluster/setup-cluster.sh | ||
|
||
#Setup Hibench | ||
source $GITHUB_WORKSPACE/actions/test-cluster/setup-hibench.sh | ||
|
||
echo "=========================================" | ||
echo "Cluster Testing with Spark Version: $SPARK_VERSION" | ||
echo "=========================================" | ||
|
||
# run all examples | ||
source $GITHUB_WORKSPACE/actions/run_all_part_2.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
#!/usr/bin/env bash | ||
|
||
# exit when any command fails | ||
set -e | ||
|
||
export HADOOP_VERSION=3.2.1 | ||
export SPARK_VERSION=3.0.0 | ||
export SPARK_BIN_VERSION=spark3.0 | ||
export SPARK_HADOOP_VERSION=hadoop3.2 | ||
export HIVE_VERSION=3.0.0 | ||
|
||
# keep track of the last executed command | ||
trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG | ||
# echo an error message before exiting | ||
trap 'echo "\"${last_command}\" command filed with exit code $?."' EXIT | ||
|
||
# mvn build | ||
mvn clean package -q -Dmaven.javadoc.skip=true -Dspark=3.0 -Dscala=2.12 | ||
|
||
# Setup cluster contain of hadoop and spark | ||
source $GITHUB_WORKSPACE/actions/test-cluster/setup-cluster.sh | ||
|
||
#Setup Hibench | ||
source $GITHUB_WORKSPACE/actions/test-cluster/setup-hibench.sh | ||
|
||
echo "=========================================" | ||
echo "Cluster Testing with Spark Version: $SPARK_VERSION" | ||
echo "=========================================" | ||
|
||
# run all examples | ||
source $GITHUB_WORKSPACE/actions/run_all_part_2.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
#!/usr/bin/env bash | ||
|
||
# exit when any command fails | ||
set -e | ||
|
||
export HADOOP_VERSION=3.2.1 | ||
export SPARK_VERSION=3.1.1 | ||
export SPARK_BIN_VERSION=spark3.1 | ||
export SPARK_HADOOP_VERSION=hadoop3.2 | ||
export HIVE_VERSION=3.0.0 | ||
|
||
# keep track of the last executed command | ||
trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG | ||
# echo an error message before exiting | ||
trap 'echo "\"${last_command}\" command filed with exit code $?."' EXIT | ||
|
||
# mvn build | ||
mvn clean package -q -Dmaven.javadoc.skip=true -Dspark=3.1 -Dscala=2.12 | ||
|
||
# Setup cluster contain of hadoop and spark | ||
source $GITHUB_WORKSPACE/actions/test-cluster/setup-cluster.sh | ||
|
||
#Setup Hibench | ||
source $GITHUB_WORKSPACE/actions/test-cluster/setup-hibench.sh | ||
|
||
echo "=========================================" | ||
echo "Cluster Testing with Spark Version: $SPARK_VERSION" | ||
echo "=========================================" | ||
|
||
# run all examples | ||
source $GITHUB_WORKSPACE/actions/run_all_part_2.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
hadoop | ||
spark |
Oops, something went wrong.