Ordering of input files nondeterministic, which can assign incorrect doc id, metadata

[dimroc]

for i, item in enumerate(list(input_path.iterdir())): can return the files in all sorts of ordering styles. I'm not even sure what it is on my local, but it isn't lexical. What ordering are you expecting?

As a result, we don't know what order to pass in the doc_ids and metadata. Let's either come up with predictable ordering or map filenames to ids/metadata so it isn't left to mismapping 0...N indices.

I have some bandwidth, so once we have a direction, I can implement it if you'd like.

Snippet:

byaldi/byaldi/colpali.py

Lines 278 to 289 in 427a859

	items = list(input_path.iterdir())
	if doc_ids is not None and len(doc_ids) != len(items):
	raise ValueError(
	f"Number of doc_ids ({len(doc_ids)}) does not match number of documents ({len(items)})"
	)
	if metadata is not None and len(metadata) != len(items):
	raise ValueError(
	f"Number of metadata entries ({len(metadata)}) does not match number of documents ({len(items)})"
	)
	for i, item in enumerate(items):
	print(f"Indexing file: {item}")
	doc_id = doc_ids[i] if doc_ids else self.highest_doc_id + 1

[bclavie]

Hey, thank you for raising, well spotted!

This has actually been on my to-do in some form, and the way I've been thinking about handling this is:

• Allow doc ids to be strings (requiring a mapping update, but that's fine) for extra flexibility
• If passing a folder, require the user passed doc ids and metadata to be a dictionary, in the format {filename: XXX}.
• Should the user not pass doc_ids, the document IDs would just default to the file names.

I'm not sure I'll have the bandwidth to do it today (I'd love to though), so if you're interested in taking this on and think you can do it quickly, you're more than welcome!

[bclavie]

In the meantime, I'm releasing 0.0.2, which among other minor updates, makes index(), add_to_index() and the newly added get_doc_ids_to_file_names() return a dict mapping {doc_id: filename}, so it's easy to pass the right context to your LLM, though it doesn't fix the metadata issue.