How do merge the results of multiple RecursiveWebLoaders? #6703
Replies: 1 comment
-
Hey there, @shreyanmitra! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this challenge together! To merge the results of multiple
Here is an example code snippet demonstrating this process: import { RecursiveUrlLoader } from '@langchain/community/document_loaders/web/recursive_url';
// Define the URLs you want to load
const urls = [
'https://example.com',
'https://another-example.com',
// Add more URLs as needed
];
// Create an array of RecursiveUrlLoader instances
const loaders = urls.map(url => new RecursiveUrlLoader(url, {
maxDepth: 2,
timeout: 10000,
preventOutside: true,
}));
// Load documents from each loader
const loadDocuments = async () => {
const allDocs = [];
for (const loader of loaders) {
const docs = await loader.load();
allDocs.push(...docs);
}
return allDocs;
};
// Use the merged documents as context for RAG
loadDocuments().then(mergedDocs => {
// mergedDocs now contains the content from all specified URLs
console.log(mergedDocs);
// You can now use mergedDocs as context for your RAG process
}); This code will load the documents from each specified URL, merge them into a single array, and then you can use this merged array as context for your RAG process. Note that the |
Beta Was this translation helpful? Give feedback.
-
Checked other resources
Commit to Help
Example Code
NA
Description
See discussion #4231
Given that there is no way to pass in an array of URLs, how do I merge a group of RecursiveBaseURL results?
My use case is based on the example at https://js.langchain.com/v0.2/docs/tutorials/rag/
I want to expand this example to perform RAG by taking the content of multiple websites as context.
Any help would be greatly appreciated.
System Info
NA
Beta Was this translation helpful? Give feedback.
All reactions