Stream files line by line when parsing to avoid memory limits

[lasalvavida]

These changes allow nedb to load large databases without running into the string length limit for node by streaming the file line by line instead of trying to load it all at once.

Tested with a large randomly generated ~256MB database that will load after these changes.

I don't know what the browser-based implications here are since this does remove Persistence.prototype.treatRawData, but I'd be happy to help resolve them if you could point me in the right direction.

[pi0]

Merged into nedb-core

[gschier]

Any chance this will be merged into NeDB?

[pi0]

@gschier Any reason why not to switch nedbhq releases at this time? it is fully compatible + many more fixes and we will try to keep up to date with upstream if actively maintained again :)

[gschier]

@pi0 I took a look but couldn't figure out whether it was affiliated with this project or not, how it differed, or what the purpose of it was. An introduction section on the README explaining purpose, roadmap, etc would be very useful! 👍

[JamesMGreene]

Seems like a symmetrical method for writing as a stream would also be necessary for the same reason.

[lasalvavida]

@JamesMGreene, shouldn't be. Pretty sure nedb writes in append mode, hence why it is possible to generate a large database that nedb cannot close and reopen.

[JamesMGreene]

@lasalvavida Not quite accurate. Although NeDB does do its updates in append mode, it also does a full datafile rewrite at the end of the initial loading process and during every compaction operation.

[JamesMGreene]

To be more clearly linked, we believe this would fix:

• how to handle if a db size larger then 256mb? #363
• Over 256Mb datafile throw err upon opening #389

[FightSCJP]

Will it be merged into nedb? What is the conclusion? How should the user solve this problem? Abandon nedb to seek other database solutions?

[xeoshow]

Hope this feature could be merged into the master branch...

[huynguyend191]

How actually can I load 1Gb of data. I use but it's not working
var Datastore = require('nedb'); var db = new Datastore({ filename: './data.db', autoload: true });