Is langchain broken with current version of node-llama-cpp?

[PeterTucker]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";
import { HumanMessage } from "@langchain/core/messages";

// Can confirm path is correct, but none of these models work
let llamaPath = "./hf_TheBloke_mistral-7b-instruct-v0.2.Q5_K_M.gguf";
llamaPath = "./hf_mradermacher_Mistral-Nemo-Instruct-2407.Q4_K_M.gguf";
llamaPath = "./Mistral-Nemo-Instruct-2407-Q4_K_M.gguf";
llamaPath = "./Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf";

const model = new ChatLlamaCpp({ modelPath: llamaPath });

const response = await model.invoke([
  new HumanMessage({ content: "My name is John." }),
]);
console.log({ response });

Description

https://js.langchain.com/docs/integrations/chat/llama_cpp/

import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";
import { HumanMessage } from "@langchain/core/messages";

// Can confirm path is correct, but none of these models work
let llamaPath = "./hf_TheBloke_mistral-7b-instruct-v0.2.Q5_K_M.gguf";
llamaPath = "./hf_mradermacher_Mistral-Nemo-Instruct-2407.Q4_K_M.gguf";
llamaPath = "./Mistral-Nemo-Instruct-2407-Q4_K_M.gguf";
llamaPath = "./Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf";

const model = new ChatLlamaCpp({ modelPath: llamaPath });

const response = await model.invoke([
  new HumanMessage({ content: "My name is John." }),
]);
console.log({ response });

Error:

TypeError: Cannot destructure property '_llama' of 'undefined' as it is undefined.
    at new LlamaModel (file:///C:/Users/User/Project/langchain-test/node_modules/node-llama-cpp/dist/evaluator/LlamaModel/LlamaModel.js:42:144)
    at createLlamaModel (file:///C:/Users/User/Project/langchain-test/node_modules/@langchain/community/dist/utils/llama_cpp.js:13:12)
    at new LlamaCpp (file:///C:/Users/User/Project/langchain-test/node_modules/@langchain/community/dist/llms/llama_cpp.js:87:23)
    at file:///C:/Users/User/Project/langchain-test/src/server.js:15:17

dependencies from package.json:

    "dependencies": {
        "@langchain/community": "^0.3.6",
        "@langchain/core": "^0.3.13",
        "node-llama-cpp": "^3.1.1"
    },

System Info

[email protected] | MIT | deps: 12 | versions: 297
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.3.2.tgz
.shasum: aec3e679d3d6c36f469448380affa475c92fbd86
.integrity: sha512-kd2kz1cS/PIVrLEDFlrZsAasQfPLbY1UqCZbRKa3/QcpB33/n6xPDvXSMfBuKhvNj0bjW6MXDR9HZTduXjJBgg==
.unpackedSize: 2.9 MB

dependencies:
@langchain/openai: >=0.1.0 <0.4.0 openapi-types: ^12.1.3
@langchain/textsplitters: >=0.0.0 <0.2.0 p-retry: 4
js-tiktoken: ^1.0.12 uuid: ^10.0.0
js-yaml: ^4.1.0 yaml: ^2.2.1
jsonpointer: ^5.0.1 zod-to-json-schema: ^3.22.3
langsmith: ^0.1.56-rc.1 zod: ^3.22.4

maintainers:

• nfcampos [email protected]
• jacoblee93 [email protected]
• andrewnguonly [email protected]
• davidduong [email protected]
• hwchase17 [email protected]
• basproul [email protected]

dist-tags:
latest: 0.3.2 tag-for-publishing-older-releases: 0.2.20
next: 0.3.2-rc.0

published a month ago by jacoblee93 [email protected]

[dosubot[bot]]

There is a similar open issue titled "LlamaCpp throwing 'TypeError: Cannot destructure property '_llama' of 'undefined' as it is undefined.'" which might be related to the problem you're experiencing. The suggested solution is to ensure that the model is in a compatible .bin file format, as the node-llama-cpp library might have issues with .gguf files. Additionally, make sure you have the node-llama-cpp module installed with a minimum version of 2.0.0 and verify that the model binary is compatible with the expected input and output formats in LangChain.js ^[1].

In your case, it seems like the models you're trying to use are in the .gguf format, which might be causing the issue. Try converting your model to a .bin format and see if that resolves the problem.

To continue talking to Dosu, mention @dosu.

[PeterTucker]

Answer from Langchain dev, "use version 2 not 3", packag.json:

    "dependencies": {
        "node-llama-cpp": "^2"
    },