Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmarks #2035

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Update benchmarks #2035

wants to merge 23 commits into from

Conversation

ZuseZ4
Copy link
Member

@ZuseZ4 ZuseZ4 commented Aug 9, 2024

Splitting this out of the rust-benchmarks to make it easier to upstream rust stuff on it's own.

closes #1414

@ZuseZ4 ZuseZ4 requested review from wsmoses and tgymnich August 9, 2024 23:20
@ZuseZ4 ZuseZ4 enabled auto-merge August 9, 2024 23:31
@ZuseZ4 ZuseZ4 disabled auto-merge August 10, 2024 03:21
@ZuseZ4
Copy link
Member Author

ZuseZ4 commented Sep 5, 2024

@wsmoses


CMake Error at /usr/lib/llvm-16/lib/cmake/llvm/AddLLVM.cmake:965 (add_executable):
  Target "enzyme-tblgen" links to target "zstd::libzstd_shared" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/lib/llvm-16/lib/cmake/llvm/TableGen.cmake:146 (add_llvm_executable)
  tools/enzyme-tblgen/CMakeLists.txt:9 (add_tablegen)

and

➜  Enzyme git:(update-benchmarks) rg find_package . 
./enzyme/CMakeLists.txt
69:find_package(LLVM REQUIRED CONFIG)
188:    find_package(Clang REQUIRED CONFIG PATHS ${Clang_DIR} NO_DEFAULT_PATH)
200:    find_package(MLIR REQUIRED CONFIG PATHS ${MLIR_DIR} NO_DEFAULT_PATH)

./enzyme/test/test_find_package/CMakeLists.txt
5:find_package(Enzyme REQUIRED CONFIG NO_DEFAULT_PATH)

./diff.txt
13:@@ -71,14 +72,14 @@ find_package(LLVM REQUIRED CONFIG)

./enzyme/test/CMakeLists.txt
16:add_custom_target(test-cmake COMMAND rm -rf ${CMAKE_CURRENT_BINARY_DIR}/test_cmake && mkdir ${CMAKE_CURRENT_BINARY_DIR}/test_cmake && cd ${CMAKE_CURRENT_BINARY_DIR}/test_cmake &&  ${CMAKE_COMMAND} -S ${CMAKE_CURRENT_BINARY_DIR}/test_cmake ${CMAKE_CURRENT_SOURCE_DIR}/test_find_package -DEnzyme_DIR=${CMAKE_BINARY_DIR} -DCMAKE_C_COMPILER=$<TARGET_FILE:clang> && ${CMAKE_COMMAND} --build ${CMAKE_CURRENT_BINARY_DIR}/test_cmake DEPENDS ClangEnzymeFlags LLDEnzymeFlags)
18:add_custom_target(test-cmake COMMAND rm -rf ${CMAKE_CURRENT_BINARY_DIR}/test_cmake && mkdir ${CMAKE_CURRENT_BINARY_DIR}/test_cmake && cd ${CMAKE_CURRENT_BINARY_DIR}/test_cmake &&  ${CMAKE_COMMAND} -S ${CMAKE_CURRENT_BINARY_DIR}/test_cmake ${CMAKE_CURRENT_SOURCE_DIR}/test_find_package -DEnzyme_DIR=${CMAKE_BINARY_DIR} -DCMAKE_C_COMPILER=$<TARGET_FILE:clang> && ${CMAKE_COMMAND} --build ${CMAKE_CURRENT_BINARY_DIR}/test_cmake DEPENDS ClangEnzymeFlags)
➜  Enzyme git:(update-benchmarks) rg zstd
enzyme/WORKSPACE
39:    name = "llvm_zstd",
40:    build_file = "@llvm-raw//utils/bazel/third_party_build:zstd.BUILD",
42:    strip_prefix = "zstd-1.5.2",
44:        "https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz"

I won't look further into such cmake issues, but if you're ok with merging it despite those (or fixing them yourself), I can fix the unsupported flag combinations. As you can see it's already broken in the other PR where I just enable the benchmarks they are already broken, so fixing at least some things is IMHO a strict improvement.

@wsmoses
Copy link
Member

wsmoses commented Sep 5, 2024

Looks like an issue in setup for the script, can you copy whatever config from other CI ensures zlib is installed?

@ZuseZ4
Copy link
Member Author

ZuseZ4 commented Sep 9, 2024

The current status is that no benchmark CI is working: https://github.com/EnzymeAD/Enzyme/actions/runs/10607142537/job/29399074787?pr=2056
With this PR at least 4 Benchmarks are run under llvm-16, so I'd recommend to merge it.
Happy to disable the still broken benchmarks, so someone else can pick them up once they have time.

@ZuseZ4 ZuseZ4 mentioned this pull request Sep 9, 2024
@ZuseZ4 ZuseZ4 requested a review from wsmoses September 16, 2024 01:09
@ZuseZ4
Copy link
Member Author

ZuseZ4 commented Sep 16, 2024

lmk how to disable the benchmarks I haven't touched and which are thus still broken and this is good to merge and should even pass CI for 15 and 16.

Copy link
Member

@wsmoses wsmoses left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So as is this removes the functionality for running the enzyme ablation hypothesis (by using the unified clang command to simplify rust linking).

can you add say another target for the reference pipeline which runs no optimization before enzyme AD?

@ZuseZ4
Copy link
Member Author

ZuseZ4 commented Sep 18, 2024

It .. doesn't, can you check again, or point out where exactly it uses ClangEnzyme? :D
This should be cosmetic and just makes sure that these tests here at least build again.
It still uses LLVM-Enzyme for opt.
I sometimes define LOADCLANG="%loadClangEnzyme", because the RustBench will use this for optimal C++ perf, but it's not used in this PR. I can remove it from this PR and reintroduce it in the next PR that includes Rust benches if you want?

I assume that merging Rustbenches will require more discussions than I have time/energy before the deadline, so this is just a fix-mvp to reduce the diff between what I use for benching and what enzyme has.

@wsmoses
Copy link
Member

wsmoses commented Sep 18, 2024

Ah it looks like it was just in a second makefile (I thought it was a sed script which modified this in place):

opt $^ $(LOAD) -enzyme -o $@ -S

Yeah this one should be modified too or ideally merged into the same makefile

@jedbrown
Copy link
Collaborator

I rebased this branch and will let CI run.

@jedbrown
Copy link
Collaborator

jedbrown commented Oct 30, 2024

I see the Bazel job is failing on main too. It's when building LLVM so must be unrelated to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants