Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting up a non-development runtime environment as a published .net application. #939

Open
AgentSmithers opened this issue Oct 4, 2024 · 11 comments

Comments

@AgentSmithers
Copy link

Description

Greetings everyone, I’ve done about a good six months of development diving into LlamaSharp including Microsoft's Kernel for rag. | validated my App works on two separate dev boxes using CUDA, CPU and Vulkan ATI but the challenge I am having is moving my application to a nondev machine via a published application. I got .Net core 8.0 runtime and SDK installed and upon running my binary | get an error stating that I don’t have a CPU backend installed... reading around the LlamaCPP and LlamaSharp(here) GitHub issues I didn't spot anything that stuck out when it came to the deployment of applications outside of a dev environment but I did see a few post recommending OpenBlas or CLBLAS as a CPU backend that involved recompiling LlamaCPP which I am not sure if that’s actually required for a LlamaSharp deployment but I can’t say for certain because on my dev box ljust run install-Package Llama.backend.cpu and everything works from there and I’m not sure if that downloads and configures OpenBlas behind the scenes. Can anyone confirm if I am going down the correct road or if I should be looking into something else? Any referenced links are also appreciated. I think my main struggle is finding detailed documentation on setting up a runtime environment vs. a dev environment. Thank you everyone in advanced!

@martindevans
Copy link
Member

There have been issue with publishing final apps before - which haven't been fixed yet. They seem to be related to the location of the binaries being different in the development app and the final built app.

Perhaps try adding a logger to the loading system (NativeLibraryConfig.All.WithLogger) and printing that out in your built application. It should show the exact paths of the DLLs it is trying to load, and you can cross check those with where the files actually are on disk.

@AgentSmithers
Copy link
Author

Thank you, this is extremely helpful to get some feedback. I thought I was doing something wrong. So theres no need to download OpenBlas or CL then to get this thing to fly, correct? It's all in ensuring the DLL's are deployed correctly and located in the right folder?
Is there an action item where I need to take a qualifying LLama.dll and rename it to LibLlama.dll? and if so does that need to be placed in the same directory as the binary or one of its sub directories? Thanks again for your feedback, I have a route to explore now.

@martindevans
Copy link
Member

martindevans commented Oct 4, 2024

In theory this should all just work ™️, so I can't give you a definite answer to what you'll need to do exactly. I'm fairly sure you won't need to rename it to libllama.dll at least, that's a Linux naming convention that's not needed for Windows DLLs.

Once you've done the logging thing if you post your results here I should be able to tell you exactly what to copy where, and after that maybe we can work out why the files aren't being deployed to the correct locations automatically :)

@AgentSmithers
Copy link
Author

AgentSmithers commented Oct 5, 2024

Runtime box is on 2019 Server CORE, not gui.

Screenshot 2024-10-05 150427
Screenshot 2024-10-05 143147

I added this to the start of my project, have confirmed x64 vc2015 runtime has been installed.
NativeLibraryConfig.All.WithLogCallback((level, message) =>
{
Console.Write($"[llama {level}]: {message}");
});

Here's a screenshot of what I have so far, on the left is my working Dev box... and on the right, the Runtime box running through RDP. The DLL reflects that the path is located but does not give any feedback as to why llama.dll fails to load.
I'm going to continue experimenting, maybe dumpbin will reflect the underlaying reason why.

update: No dice, dumpbin reflected VC140*.dll's which is why I figured maybe the runtime was missing for one of the dependencies. Still receiving the same error.

@martindevans
Copy link
Member

martindevans commented Oct 6, 2024

Ah I initially assumed this was an issue with the DLLs simply being in the wrong place, but that seems to be totally wrong! Failed to load means NativeLibrary.TryLoad was called (a runtime method) but the runtime failed to load the DLL.

Dumpbin would have been my next suggestion, but you beat me to it! There was nothing else missing besides the vc140 runtime?

@AgentSmithers
Copy link
Author

AgentSmithers commented Oct 6, 2024

Correct, not as far as I could tell, I ran dumpbin /dependents and spotted that VC140's DLLs were listed but after installing VC++ 2015 Runtime both for x64 and x86, same error. Dumpbin didn't say anything explicit about anything missing from the system, is there a preferred command to run to confirm this that I may not be aware of? I'm open to any other suggestions.

@martindevans
Copy link
Member

I'm a bit stumped by that I'm afraid. The only other thing I can think of is AVX support, but we distribute a DLL with no AVX requirements at all to account for that and it failed to load as well!

@AgentSmithers
Copy link
Author

Any idea on what method can be used to find the underlaying loading issues?
I trued fusion but it did not display anything in the logs and am not sure if It should reflect their or if I need to try some other method to get to the root cause of the DLL failing.

@martindevans
Copy link
Member

A few ideas:

  • You could try dependency walker
  • Sometimes when you download a file windows marks it as blocked, and won't allow it to be opened. Maybe check that for the DLLs?
  • Check the file hash to make sure it's not been corrupted during download.
  • You could try cloning llama.cpp and compiling the binaries on the target machine (make sure you use exactly the correct version, there is no version-to-version compatibility).

@AgentSmithers
Copy link
Author

AgentSmithers commented Oct 10, 2024

So I ran dependacy walker. Found that VCRUNTIME140_1.dll was noted to be missing.
So I visited:
https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170
and downloaded and installed ALL of the runtimes on my machine, both x86 and x64.

DW

Now I am receiving this. Seems that even after installing a 2005 redist, that maybe perhaps the 2015-2019 vcruntime was also required and I am now receiving a different error and will continue to troubleshoot.

Screenshot 2024-10-10 094219

update I got it, I had to go back to Visual Studio and compile just for windows and have validated its operating on server 2019 core setup.

Screenshot 2024-10-10 094442

Thanks for the suggestion with Depends! It helps point out issues with DLLs where dumpbin does not (AFAIK).

I will add, oddly enough after deploying the 8.0-Windows version of the binary, Depends still reflects that the same DLL is missing, so maybe I just guessed that the VCRuntimes needed to be installed by accident. Eh, in anyevent I'm up and running.

Screenshot 2024-10-10 100739

@zsogitbe
Copy link
Contributor

This is exactly one of the reasons why you should compile the code yourself and don't use pre-compiled packages. Even if you find the missing DLLs now, the problem could occur again later with different DLLs when the packages is recompiled with slightly different settings. Also for security reasons try to compile as much code yourself as possible. If you compile it yourself, then you will be sure about the needed DLLs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants