Tracking issue: metadata filtering

[asg017]

sqlite-vec doesn't have good metadata filtering as of v0.1.0. Only vector columns can be declared in the vec0 constructor. You can do pre-filtering with vec_column IN (...) queries, but that's slow and inconvenient.

I'm thinking:

create virtual table vec_movies(
  movie_id text primary key,
  genre text,
  release_date date,
  rating text,
  is_3d boolean,
  synopsis_embedding float[768]
);

genre, release_date, rating, and is_3d would all be "metadata" columns. You could do queries like:

select
  rowid,
  distance
from vec_movies
where synopsis_embedding match embed('comedic american summer camp')
  and k = 20
  and is_3d
  and release_date between '2010-01-01' and '2015-12-31'
  and rating = 'PG';

We could capture all the WHERE clauses to ensure that the top 20 returned vectors match that criteria.

A few open questions:

How do we store metadata values?

We could store in OLTP-fashion with the _rowids shadow tables, but that may be slow. We could store in column-oriented fashion to match the vector column formats, but unsure how much faster that would be.

How would this work with ANN indexes?

🤷

What datatypes to support?

Ideally everything, ideally STRICT. But if we do column-oriented we'd need a strict subset. like:

• TEXT
• INT
• DOUBLE
• BLOB
• BOOLEAN
• DATE/DATETIME

Maybe we could do dictionary encoding for text values? maybe that's a column option, like genre text encoding=dictionary or something. Maybe ENUMs? NULL/NOT NULL?

[forrestbao]

I really wish this feature can be available soon.

[ajram23]

@asg017 +1. Looks like langchain expects the metadata to be available as a dictionary. I have tried the integration and this is the last remaining piece to migrate fully out of ChromaDB.

[lojik-ng]

+1