Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add SYCL feature flags to rllm-llamacpp build (To add support for Intel GPUs) #96

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

TheMrCodes
Copy link

The goal of this pull request is to add support for Intel GPU to the build script of rllm llama.cpp
This is done by

  • expanding the cargo feature flags of rllm-llamacpp and rllm-llamacpp_low and incoporate
  • adding the needed build arguments in the main.rs from rllm-llamacpp_low
  • and adding appropriate new options to server.sh

The code will be tested on a machine with Intel 13Gen Processor & Intel Arc A770 GPU

@TheMrCodes
Copy link
Author

@microsoft-github-policy-service agree

Comment on lines 49 to 89

let dirs = [
"/opt/intel/oneapi/compiler/latest/lib",
"/opt/intel/oneapi/mkl/latest/lib",
//"/opt/intel/oneapi/dnnl/latest/lib",
];

// *.a => static
// *.so => dynamic
for dir in dirs.iter() {
println!("cargo:rustc-link-search={}", dir);
for file in std::fs::read_dir(dir).unwrap() {
let file = file.unwrap();
let file_name = file.file_name();
let file_name = file_name.to_str().unwrap();
if !file_name.starts_with("lib") { continue; }
if file_name.contains("lp64") && !file_name.contains("ilp64") { continue; }
if file_name.contains("seq") { continue; }
if file_name == "libmkl_gnu_thread.so" { continue; }
let file_name = file_name.trim_start_matches("lib");

if file_name.ends_with(".so") {
let file_name = &file_name[..file_name.len()-3];
println!("cargo:rustc-link-lib=dylib={}", file_name);
} else if file_name.ends_with(".a") {
let file_name = &file_name[..file_name.len()-2];
println!("cargo:rustc-link-lib=static={}", file_name);
}
}
}
//panic!("stop here");

//println!("cargo:rustc-link-search=native=/opt/intel/oneapi/compiler/latest/lib");
//println!("cargo:rustc-link-lib=intlc");
//println!("cargo:rustc-link-lib=svml");
//println!("cargo:rustc-link-lib=sycl");
//println!("cargo:rustc-link-search=native=/opt/intel/oneapi/mkl/latest/lib");
//println!("cargo:rustc-link-lib=mkl_core");
//println!("cargo:rustc-link-lib=mkl_sycl_blas");
//println!("cargo:rustc-link-lib=mkl_sycl");
}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently struggling with dynamically linken the libraries but also resolving only needed without duplicates and with resolved dependencies. Don't really know whats a good way here...

@TheMrCodes
Copy link
Author

The basic build works now. 🚀
Want to test the feature again if the llama.cpp version got bumped

Known Issues:

  • llama.cpp hangs after loading model into VRAM on Intel ARC -> This is a bug of the current llama.cpp version and has to be tested again after an upgrade
  • currently only FP32 build works -> This also could depend on the current version but further testing has to be done

@TheMrCodes TheMrCodes marked this pull request as ready for review April 23, 2024 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant