New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creating an HNSW index causes severe data inflation #32

Open

QuanZi123 opened this issue Oct 22, 2024 · 0 comments

QuanZi123 commented Oct 22, 2024 •

edited

Loading

duckdb version : 1.1.1

When create HNSW indexes first and then insert data, duckdb's data files are 100 times larger than the original files

But if insert the data first and then create the HNSW index, duckdb's data swells by about 15 times

Test data and code duckdb_index

Here's my test result:

Original file train.jsonl : 78371kb

File speed_3.db (create index first) : 7055628KB

File speed_t.db (insert data first) : 1328369KB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment