Skip to content

Commit

Permalink
community[patch]: fixed multithreading returning List[List[Documents]…
Browse files Browse the repository at this point in the history
…] instead of List[Documents] (#20230)

Description: When multithreading is set to True and using the
DirectoryLoader, there was a bug that caused the return type to be a
double nested list. This resulted in other places upstream not being
able to utilize the from_documents method as it was no longer a
`List[Documents]` it was a `List[List[Documents]]`. The change made was
to just loop through the `future.result()` and yield every item.
Issue: #20093
Dependencies: N/A
Twitter handle: N/A
  • Loading branch information
chip-davis authored Apr 9, 2024
1 parent 230376f commit 806d4ae
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,8 @@ def lazy_load(self) -> Iterator[Document]:
)
)
for future in concurrent.futures.as_completed(futures):
yield future.result()
for item in future.result():
yield item
else:
for i in items:
yield from self._lazy_load_file(i, p, pbar)
Expand Down

0 comments on commit 806d4ae

Please sign in to comment.