Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run benchmarks weekly in CI #1245

Open
wants to merge 21 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5110a63
Create run benchmarks script and setup in CI
chapman39 Oct 9, 2024
f1e6916
Use environment variables for host configs and cmake args
chapman39 Oct 9, 2024
a6702b1
find host config path
chapman39 Oct 9, 2024
7c95a15
Mention path to shared caliper files
chapman39 Oct 9, 2024
1a1097b
increase ci benchmark alloc time
chapman39 Oct 9, 2024
0e93214
Check if petsc is enabled before running petsc multigrid
chapman39 Oct 10, 2024
8849962
return result of benchmarks
chapman39 Oct 10, 2024
77706dd
increase deadline time to allow for longer jobs to stay on the queue …
chapman39 Oct 10, 2024
e5af250
Merge remote-tracking branch 'origin/develop' into feature/chapman39/…
chapman39 Oct 25, 2024
5d266ef
Ensure all cray jobs are disabled until all developers have access
chapman39 Oct 25, 2024
d0fd0c3
increase alloc time on benchmarks now that they take longer
chapman39 Oct 25, 2024
cbcc38f
add alloc_deadline variable to all toss4 jobs
chapman39 Oct 25, 2024
9d95042
Merge branch 'develop' into feature/chapman39/run-benchmarks-ci
chapman39 Oct 28, 2024
f337dde
build benchmarks as release
chapman39 Oct 28, 2024
dd05693
Merge branch 'feature/chapman39/run-benchmarks-ci' of github.com:LLNL…
chapman39 Oct 28, 2024
faf3ff9
increase benchmark ctests to be 1.5 hours per job
chapman39 Oct 29, 2024
cfd2d33
Merge branch 'develop' into feature/chapman39/run-benchmarks-ci
chapman39 Oct 29, 2024
2ba2921
comment
chapman39 Oct 29, 2024
1853080
Merge branch 'feature/chapman39/run-benchmarks-ci' of github.com:LLNL…
chapman39 Oct 29, 2024
935cfbb
Merge branch 'develop' into feature/chapman39/run-benchmarks-ci
chapman39 Oct 29, 2024
9c1b3e4
set property to actual test not exec
chapman39 Oct 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@ variables:

.src_workflow:
rules:
- if: '$FULL_BUILD != "ON"'
- if: '$FULL_BUILD != "ON" && $BENCHMARKS_BUILD != "ON"'

.full_workflow:
rules:
- if: '$FULL_BUILD == "ON"'
- if: '$FULL_BUILD == "ON" && $BENCHMARKS_BUILD != "ON"'

.benchmarks_workflow:
rules:
- if: '$FULL_BUILD != "ON" && $BENCHMARKS_BUILD == "ON"'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could alternatively use some sort of CI_WORKFLOW_TYPE var which could be src, full, or benchmarks. That way we don't have to handle the two variables. I just don't have access to nightly build settings.


####
# Templates
Expand Down Expand Up @@ -66,6 +70,22 @@ variables:
reports:
junit: ${FULL_BUILD_ROOT}/${SYS_TYPE}/*/_serac_build_and_test_*/build-*/junit.xml


.benchmarks_build_script:
script:
# Builds src, runs benchmarks, and stores Caliper files in shared location
- echo -e "section_start:$(date +%s):benchmarks_build\r\e[0K
Benchmarks Build ${CI_PROJECT_NAME}"
- ${ALLOC_COMMAND} python3 scripts/llnl/run_benchmarks.py
- echo -e "section_end:$(date +%s):benchmarks_build\r\e[0K"
artifacts:
expire_in: 2 weeks
when: always
paths:
- _serac_build_and_test_*/output.log*.txt
- _serac_build_and_test_*/build-*/output.log*.txt
- _serac_build_and_test_*/build-*/*.cali

# This is where jobs are included for each system
include:
- local: .gitlab/build_blueos.yml
Expand Down
12 changes: 12 additions & 0 deletions .gitlab/build_blueos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@
extends: [.full_build_script, .on_blueos, .full_workflow]
needs: []

.benchmarks_build_on_blueos:
extends: [.benchmarks_build_script, .on_blueos, .benchmarks_workflow]
needs: []

####
# Build jobs
blueos-clang_10_0_1-src:
Expand All @@ -58,3 +62,11 @@ blueos-clang_10_0_1-full:
ALLOC_NODES: "1"
ALLOC_TIME: "55"
extends: [.full_build_on_blueos, .with_cuda]

blueos-clang_10_0_1-benchmarks:
variables:
COMPILER: "[email protected]"
HOST_CONFIG: "lassen-blueos_3_ppc64le_ib_p9-${COMPILER}_cuda.cmake"
ALLOC_NODES: "1"
ALLOC_TIME: "30"
extends: [.benchmarks_build_on_blueos, .with_cuda]
22 changes: 20 additions & 2 deletions .gitlab/build_toss4.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@
# LC version of pip is ancient
- if [[ $(python3 -c 'import pip; print(pip.__version__ < "19.3")') == "True" ]]; then python3 -m pip install --user --upgrade pip; fi

.benchmarks_build_on_toss4:
extends: [.benchmarks_build_script, .on_toss4, .benchmarks_workflow]
needs: []


####
# Build jobs
Expand Down Expand Up @@ -76,7 +80,6 @@ toss4-clang_14_0_6-full:
SPEC: "--spec=%${COMPILER}"
ALLOC_NODES: "1"
ALLOC_TIME: "45"
EXTRA_CMAKE_OPTIONS: "-DENABLE_BENCHMARKS=ON"
extends: .full_build_on_toss4

toss4-gcc_10_3_1-full:
Expand All @@ -85,5 +88,20 @@ toss4-gcc_10_3_1-full:
SPEC: "--spec=%${COMPILER}"
ALLOC_NODES: "1"
ALLOC_TIME: "45"
EXTRA_CMAKE_OPTIONS: "-DENABLE_BENCHMARKS=ON"
extends: .full_build_on_toss4

toss4-clang_14_0_6-benchmarks:
variables:
COMPILER: "[email protected]"
HOST_CONFIG: "ruby-toss_4_x86_64_ib-${COMPILER}.cmake"
ALLOC_NODES: "1"
ALLOC_TIME: "30"
extends: .benchmarks_build_on_toss4

toss4-gcc_10_3_1-benchmarks:
variables:
COMPILER: "[email protected]"
HOST_CONFIG: "ruby-toss_4_x86_64_ib-${COMPILER}.cmake"
ALLOC_NODES: "1"
ALLOC_TIME: "30"
extends: .benchmarks_build_on_toss4
15 changes: 12 additions & 3 deletions .gitlab/build_toss4_cray.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@
# LC version of pip is ancient
- if [[ $(python3 -c 'import pip; print(pip.__version__ < "19.3")') == "True" ]]; then python3 -m pip install --user --upgrade pip; fi

.benchmarks_build_on_toss4_cray:
extends: [.benchmarks_build_script, .on_toss4_cray, .benchmarks_workflow]
needs: []


####
# Build jobs
Expand All @@ -41,9 +45,14 @@ toss4_cray-clang_17_0_0-src:

toss4_cray-clang_17_0_0-full:
variables:
COMPILER: "[email protected]"
SPEC: "--spec=%${COMPILER}"
ALLOC_NODES: "1"
ALLOC_TIME: "45"
EXTRA_CMAKE_OPTIONS: "-DENABLE_BENCHMARKS=ON"
extends: .full_build_on_toss4_cray

toss4_cray-clang_17_0_0-benchmarks:
variables:
COMPILER: "[email protected]"
HOST_CONFIG: "tioga-toss_4_x86_64_ib_cray-${COMPILER}_hip.cmake"
ALLOC_NODES: "1"
ALLOC_TIME: "30"
extends: .benchmarks_build_on_toss4_cray
109 changes: 109 additions & 0 deletions scripts/llnl/run_benchmarks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/bin/sh
"exec" "python3" "-u" "-B" "$0" "$@"

# Copyright (c) 2019-2024, Lawrence Livermore National Security, LLC and
# other Serac Project Developers. See the top-level LICENSE file for details.
#
# SPDX-License-Identifier: (BSD-3-Clause)

"""
file: run_benchmarks.py

description:
Run benchmarks and update shared (or any desired) location with new Caliper files

"""

from common_build_functions import *

from argparse import ArgumentParser

import os


def parse_args():
"Parses args from command line"
parser = ArgumentParser()
parser.add_argument("-e", "--extra-cmake-options",
dest="extra_cmake_options",
default=os.environ.get("EXTRA_CMAKE_OPTIONS", ""),
help="Extra cmake options to add to the cmake configure line. Note '-DENABLE_BENCHMARKS=ON -DENABLE_DOCS=OFF' is always appended.")
parser.add_argument("-hc", "--host-config",
dest="host_config",
default=os.environ.get("HOST_CONFIG", None),
help="Specific host-config file to build (defaults to HOST_CONFIG environment variable)")
parser.add_argument("-sd", "--spot-directory",
dest="spot_dir",
default=get_shared_spot_dir(),
help="Where to put all resulting caliper files to use for SPOT analysis (defaults to a shared location)")
parser.add_argument("-t", "--timestamp",
dest="timestamp",
default=get_timestamp(),
help="Set timestamp manually for debugging")

# Parse args
args, extra_args = parser.parse_known_args()
args = vars(args)

# Verify args
if args["host_config"] is None:
print("[ERROR: Both host_config argument and HOST_CONFIG environment variable unset!]")
sys.exit(1)

if not os.path.exists(args["host_config"]):
print("[ERROR: Host config path does not exist: %s]" % args["host_config"])
sys.exit(1)

return args


def main():
# Args
args = parse_args()
cmake_options = args["extra_cmake_options"] + " -DENABLE_BENCHMARKS=ON -DENABLE_DOCS=OFF"
host_config = args["host_config"]
spot_dir = args["spot_dir"]
timestamp = args["timestamp"]

# Vars
repo_dir = get_repo_dir()
test_root = get_build_and_test_root(repo_dir, timestamp)
host_config_root = get_host_config_root(host_config)
benchmarks_output_file = os.path.join(test_root, "output.log.%s.benchmarks.txt" % host_config_root)

# Build Serac
os.chdir(repo_dir)
os.makedirs(test_root, exist_ok=True)
build_and_test_host_config(test_root=test_root, host_config=host_config,
report_to_stdout=True, extra_cmake_options=cmake_options,
skip_install=True, skip_tests=True)

# Go to build location
build_dir=""
dirs = glob.glob(pjoin(test_root, "*"))
for dir in dirs:
if os.path.exists(dir) and "build-" in dir:
build_dir=dir
os.chdir(build_dir)

# Run benchmarks
shell_exec("make run_benchmarks", echo=True, print_output=True, output_file=benchmarks_output_file)

# Move resulting .cali files to specified directory
os.makedirs(spot_dir, exist_ok=True)
cali_files = glob.glob(pjoin(build_dir, "*.cali"))
for cali_file in cali_files:
if os.path.exists(cali_file):
shutil.copy2(cali_file, spot_dir)

# Print SPOT url
if on_rz():
print("[View SPOT directory here: https://rzlc.llnl.gov/spot2/?sf={0}]".format(spot_dir))
else:
print("[View SPOT directory here: https://lc.llnl.gov/spot2/?sf={0}]".format(spot_dir))

return 0


if __name__ == "__main__":
sys.exit(main())