Dense Passage Retrieval.

[NiclaAuletta]

Hi!

A year ago I found a tutorial Training Your Own Dense Passage Retrieval Model through Haystack 1.x.
I would like to know how to do the migration of this code on Haystack 2.x.

Thanks to you all!

[Amnah199]

Hi @NiclaAuletta,

There isn’t a DPR training tutorial for Haystack 2.x yet. However, one possible approach could be to train a dense model (DPR-like) using external libraries like Sentence-Transformers. Once trained, you can use this model with an embedder and a retriever component. Below is a rough example for this approach:

from haystack import Document, Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
from haystack.components.retrievers import InMemoryEmbeddingRetriever

document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")

documents = [
    Document(content="There are over 7,000 languages spoken around the world today."),
    Document(content="Elephants have been observed to show self-awareness, e.g., recognizing themselves in mirrors."),
]

# (Replace "model_name_or_path" with the path or name of your fine-tuned model)
document_embedder = SentenceTransformersDocumentEmbedder(
    model_name_or_path="path/to/your-trained-model"
)
document_embedder.warm_up()

docs_with_embeddings = document_embedder.run(documents)["documents"]
document_store.write_documents(docs_with_embeddings)

# Build a retrieval pipeline
query_pipeline = Pipeline()

query_pipeline.add_component(
    "text_embedder", 
    SentenceTransformersTextEmbedder(model_name_or_path="path/to/your-trained-model")
)
query_pipeline.add_component(
    "retriever", 
    InMemoryEmbeddingRetriever(document_store=document_store)
)

query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")