-
-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Database Size for mobile #35
Comments
Hey @MentalGear thanks for checking out Voy! Voy doesn't support automatic partitioning or caching at the moment. Voy is designed for limited environments like mobile and edge servers! The current tree-structured index in WebAssemly is able to provide efficient traversal through your data. I think for your use case it'll perform great. But if you start to experience performance issues as your dataset grows, please do create an issue. I really like your idea about automatic sharding. I'll look into it. |
Hey, glad you like the idea! Indeed Voy is highly efficient in the retrieval! I found that overall in-Memory size per tab varies for mobile browsers quite a bit: On-Device Storage: I was wondering if Voy could integrate, besides a normal export function, storage adapters, most preferable with something like OPFS which is fast and supported by all mayor browsers now. (Reference: https://rxdb.info/rx-storage-opfs.html) |
What about using web workers? Using a shared worker might give you more freedom from the memory limitation per tab. |
Yes, I'm also interested in this, but could not find definitive resources for web workers. Shared Web workers allow the usage of the same worker across multiple tabs of the same web resource, however I don't think they expand overall available memory (but I don't know for sure). There's a few APIs, however with very limited browser support (https://developer.mozilla.org/en-US/docs/Web/API/Performance/memory). |
Hey @MentalGear I've done some more digging. Because Voy uses the embeddings as coordinates to store and locate minimal data from the resources, the requirement for the storage is greatly reduced (no embeddings are stored). For the resource data, like title, description, and url, I can provide you with an API that allows serialized and unstructured data to further minimize the size. |
Yes, adding custom data to records would be a huge plus ! |
EDITED.
Voy looks really promising, especially with it's quick retrieval and low footprint.
My use case is running it locally in the browser, to index photos, and I expect the overall database to be around 1-2 GB. Does voy provide any optimization like sharding to ensure mobile browsers are not overwhelmed (or automatic release of least used items like vector-storage ) ?
( Tool to test memory size on mobile: https://jsfiddle.net/jasonklotzer/kvnouyz7/ )
The text was updated successfully, but these errors were encountered: