Issue Using LlamaCppEmbedding with Chroma Class

[yesesuljoo]

Example Code

import { Chroma } from '@langchain/community/vectorstores/chroma';
import { AzureOpenAIEmbeddings } from '@langchain/openai';
import { LlamaCppEmbeddings } from '@langchain/community/embeddings/llama_cpp';

async getAzureEmbeddings() {
    return new AzureOpenAIEmbeddings({
        azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY,
        azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
        azureOpenAIApiDeploymentName: 'text-embedding-ada-002',
        azureOpenAIApiVersion: '2024-05-01-preview',
    });
}

async getLlamaCppEmbedding() {
    return await LlamaCppEmbeddings.initialize({
        modelPath: process.env.LLAMA_EMBEDDING_PATH,
    });
}

async getVectorStore(embeddings) {
    return new Chroma(embeddings, {
        collectionName: this.collectionName,
        url: process.env.CHROMA_SERVER_URL,
        collectionMetadata: {
            'hnsw:space': 'cosine',
        },
    });
}

async addDocWithAzureEmbedding(userDocument) {
    const azureEmbeddings = await getAzureEmbeddings();
    const vectorStore = await getVectorStore(azureEmbeddings);
    await vectorStore.addDocuments(userDocument); // Works fine with AzureOpenAIEmbeddings
}

async addDocWithLlamaCppEmbedding(userDocument) {
    const llamaEmbeddings = await getLlamaCppEmbedding();
    const vectorStore = await getVectorStore(llamaEmbeddings);
    await vectorStore.addDocuments(userDocument); // Error Occurs
}

//Test
const document1 = { pageContent: "foo", metadata: { baz: "bar" } };
const document2 = { pageContent: "thud", metadata: { bar: "baz" } };
const document3 = { pageContent: "i will be deleted :(", metadata: {} };
const documents = [document1, document2, document3];

addDocWithAzureEmbedding(documents);
addDocWithLlamaCppEmbedding(documents);

Description

I am trying to use the Chroma class from @langchain/community/vectorstores/chroma with multiple embedding functions. While testing, I noticed that it works correctly with AzureOpenAIEmbeddings, but I encounter an error when using LlamaCppEmbedding.

It seems the issue arises because the embedDocuments method in LlamaCppEmbedding returns tokenized values of the input text instead of mapped embedding vectors. For AzureOpenAIEmbeddings, the embedDocuments method always returns fixed-dimension embedding vectors for any given input, but for LlamaCppEmbedding, the dimensions of the returned vectors vary depending on the input text.

I suspect this inconsistency is causing the error when adding documents to Chroma using addDocuments. Has anyone experienced similar issues or found a workaround?

I am attaching the error message below.

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
file:///home/yeseul/work/vectordb/node_modules/chromadb/dist/chromadb.mjs:2851
throw new Error(
^

Error: Failed to fetch http://localhost:8000/api/v1/collections?tenant=default_tenant&database=default_database with status 422: Unprocessable Entity
at chromaFetch (file:///home/yeseul/work/vectordb/node_modules/chromadb/dist/chromadb.mjs:2851:13)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async ChromaClient.getOrCreateCollection
...
Node.js v18.20.5

Thanks in advance for your help!

System Info

"@langchain/community": "^0.3.16",
"@langchain/openai": "0.2.11",

[dosubot[bot]]

Hello, @yesesuljoo! I'm here to help you with any bugs, questions, or contributions you have. Let's see if we can tackle this issue together.

The issue you're encountering is likely due to the embedDocuments method in the LlamaCppEmbeddings class returning tokenized values with varying dimensions, which is not compatible with the Chroma class that expects fixed-dimension embedding vectors. The Chroma class requires consistent vector dimensions to properly store and index the documents.

To resolve this, you can ensure that the LlamaCppEmbeddings class returns fixed-dimension vectors. Here are a few potential solutions:

1. Post-Processing the Embeddings: After obtaining the embeddings from LlamaCppEmbeddings, you can post-process them to ensure they have a consistent dimension. This might involve padding the vectors with zeros or truncating them to a fixed size.

2. Configuration Options: Check if there are configuration options in the LlamaCppEmbeddings class that allow you to specify the desired output dimensions. Unfortunately, I couldn't find specific configuration options for this in the repository, but it's worth checking the documentation or source code for any such settings.

3. Custom Embedding Function: If the above options are not feasible, consider implementing a custom embedding function that wraps around LlamaCppEmbeddings to ensure the output vectors are of a consistent dimension before passing them to the Chroma class.

These approaches should help you align the output of LlamaCppEmbeddings with the requirements of the Chroma class, allowing you to add documents without encountering dimension-related errors.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}