-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stream files line by line when parsing to avoid memory limits #463
base: master
Are you sure you want to change the base?
Conversation
Merged into nedb-core |
Any chance this will be merged into NeDB? |
@pi0 I took a look but couldn't figure out whether it was affiliated with this project or not, how it differed, or what the purpose of it was. An introduction section on the README explaining purpose, roadmap, etc would be very useful! 👍 |
Seems like a symmetrical method for writing as a stream would also be necessary for the same reason. |
@JamesMGreene, shouldn't be. Pretty sure nedb writes in append mode, hence why it is possible to generate a large database that nedb cannot close and reopen. |
@lasalvavida Not quite accurate. Although NeDB does do its updates in append mode, it also does a full datafile rewrite at the end of the initial loading process and during every compaction operation. |
To be more clearly linked, we believe this would fix: |
Will it be merged into nedb? What is the conclusion? How should the user solve this problem? Abandon nedb to seek other database solutions? |
Hope this feature could be merged into the master branch... |
How actually can I load 1Gb of data. I use but it's not working |
These changes allow nedb to load large databases without running into the string length limit for node by streaming the file line by line instead of trying to load it all at once.
Tested with a large randomly generated ~256MB database that will load after these changes.
I don't know what the browser-based implications here are since this does remove
Persistence.prototype.treatRawData
, but I'd be happy to help resolve them if you could point me in the right direction.