Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run benchmarks weekly in CI #1245

Open
wants to merge 21 commits into
base: develop
Choose a base branch
from

Conversation

chapman39
Copy link
Contributor

@chapman39 chapman39 commented Oct 9, 2024

This PR...

  • Adds run_benchmarks.py script that builds Serac, runs benchmarks, and moves resulting Caliper files in shared location on LC.
  • Sets up CI such that this script is run on all platforms (ruby, lassen, tioga)
  • Setup weekly Pipeline Schedule on Gitlab

Also,

  • Disables benchmarks on full builds (since benchmarks are being compiled on src already)
  • Fixes cray full CI builds? (it was not using the spec in specs.json)
  • Fixes physics_benchmark_solid_nonlinear_solve benchmark to handle petsc being disabled
  • Disables all cray CI jobs until all developers have access (only on-push was disabled previously)

@chapman39 chapman39 added CI Continuous Integration testing Related to testing labels Oct 9, 2024
@chapman39 chapman39 self-assigned this Oct 9, 2024
@chapman39 chapman39 changed the title Create run benchmarks script and setup in CI Run benchmarks weekly in CI Oct 9, 2024

.benchmarks_workflow:
rules:
- if: '$FULL_BUILD != "ON" && $BENCHMARKS_BUILD == "ON"'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could alternatively use some sort of CI_WORKFLOW_TYPE var which could be src, full, or benchmarks. That way we don't have to handle the two variables. I just don't have access to nightly build settings.

@chapman39 chapman39 marked this pull request as ready for review October 25, 2024 21:20
@chapman39
Copy link
Contributor Author

chapman39 commented Oct 25, 2024

In a future PR I'll update the docs to explain how to properly view this data on SPOT, but it's more or less the following:

  • Go to https://lc.llnl.gov/spot2/?sf=/usr/WS2/smithdev/califiles
  • Click the check mark button on the top right
  • Ensure "cluster" is ticked on
  • Click on "ruby" under the cluster pie chart (of whatever machine you want to view)
  • Click on the benchmark you want to view under the "executable" pie chart
  • Scroll down to the table and and select the "compare" tab
  • Above the charts set "Group by" to "mpi.world.size"

You'll notice at the moment each chart is practically a straight line, because there haven't been any changes to Serac yet. If you want to view another cluster or executable, click on the previous one to deselect it before selecting a new one. That way to you don't view both at the same time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration testing Related to testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants