NestDB

The Node.js Embedded JavaScript Database

Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, written in 100% JavaScript, no binary dependency. Originally forked from NeDB. API is similar to MongoDB's and it's plenty fast.

IMPORTANT NOTE: Please don't submit issues for questions regarding your code. Only actual bugs or feature requests will be answered, all others will be closed without comment. Also, please follow the bug reporting guidelines and check the change log before submitting an already fixed bug :)

Installation

Node.js version via NPM

$ npm install nestdb --save

Node.js version via Yarn

$ yarn add nestdb

Browser version via Bower

$ bower install nestdb --save    # Main files under "./browser-version/out/"

Compatibility with NeDB

NestDB was originally forked from NeDB.

NestDB v2.x will maintain backward compatibility with NeDB v1.x to make for seamless transitions migrating from NeDB to NestDB.

When we eventually release NestDB v3.x, it will not be backward compatible with NeDB v1.x.

API

It is similar to MongoDB's API (the most used operations). The NestDB API is a superset of NeDB's API.

Creating/loading a datastore
Persistence
Inserting documents
Finding documents
Counting documents
Updating documents
Removing documents
Indexing
Destroying a datastore
Extending with plugins
Using a custom storage engine

Creating/loading a datastore

You can use NestDB as an in-memory only datastore or as a persistent datastore. One datastore is the equivalent of a MongoDB collection. The constructor is used as follows new Datastore(options) where options is an object with the following fields:

filename (optional): path to the file where the data is persisted. If left blank, the datastore is automatically considered in-memory only. It cannot end with a ~ which is used in the temporary files NestDB uses to perform crash-safe writes.
inMemoryOnly (optional, defaults to false): as the name implies.
timestampData (optional, defaults to false): timestamp the insertion and last update of all documents, with the fields createdAt and updatedAt. User-specified values override automatic generation, usually useful for testing.
autoload (optional, defaults to false): if used, the datastore will automatically be loaded from the datafile upon creation (you don't need to call load). Any command issued before load is finished is buffered and will be executed when load is done.
onload (optional): if you use autoloading, this is the handler called after the load. It takes one error argument. If you use autoloading without specifying this handler, and an error happens during load, an error will be thrown.
idGenerator (optional): if set, this function will be used for generating IDs. It takes no arguments and should return a unique string.
afterSerialization (optional): hook you can use to transform data after it was serialized and before it is written to disk. Can be used for example to encrypt data before writing datastore to disk. This function takes a string as parameter (one line of an NestDB data file) and outputs the transformed string, which must absolutely not contain a \n character (or data will be lost).
beforeDeserialization (optional): inverse of afterSerialization. Make sure to include both and not just one or you risk data loss. For the same reason, make sure both functions are inverses of one another. Some failsafe mechanisms are in place to prevent data loss if you misuse the serialization hooks: NestDB checks that never one is declared without the other, and checks that they are reverse of one another by testing on random strings of various lengths. In addition, if too much data is detected as corrupt, NestDB will refuse to start as it could mean you're not using the deserialization hook corresponding to the serialization hook used before (see below).
corruptAlertThreshold (optional): between 0 (0%) and 1 (100%), defaults to 0.1 (10%). NestDB will refuse to start if more than this percentage of the datafile is corrupt. 0 means you don't tolerate any corruption, 1 means you don't care.
compareStrings (optional): function compareStrings(a, b) should compare strings a and b and must return -1, 0 or 1. If specified, it overrides default string comparison (===), which is not well adapted to non-US characters such as accented or diacritical letters. Using the native String.prototype.localeCompare will be the right choice most of the time.
storage (optional): A custom storage engine for the database files. Must implement at least the handful of methods exported by the standard "storage" module included in NestDB, as detailed below in the Using a custom storage engine section.

If you use a persistent datastore without the autoload option, you need to call load manually. This function fetches the data from datafile and prepares the datastore. Do NOT forget it! If you use a persistent datastore, no command (e.g. insert, find, update, remove) will be executed before load is called, so make sure to either call it yourself or use the autoload option.

Also, if load fails, all commands registered to the executor afterwards will not be executed. They will be registered and executed, in sequence, only after a successful load.

Once the datastore is fully loaded, it also emits a "loaded" event that you can add a listener for.

Whenever any datastore is fully loaded, the Datastore object itself emits a global "created" event that you can add a listener for.

Type 1: In-memory only datastore (no need to load)

var Datastore = require('nestdb')
  , db = new Datastore();
// You can issue commands right away

Type 2: Persistent datastore with manual loading

var Datastore = require('nestdb')
  , db = new Datastore({ filename: 'path/to/datafile' });
db.load(function (err) {    // Callback is optional
  // Now commands will be executed
});

Type 3: Persistent datastore with automatic loading

var Datastore = require('nestdb')
  , db = new Datastore({
          filename: 'path/to/datafile'
        , autoload: true
        , onload: function (err) {
            if (err) {
              console.error('Failed to load the datastore:', err);
            } else {
              console.log('Loaded the datastore!');
            }
          }
         });
// You can issue commands right away because of NestDB's internal queueing

// You can also synchronously add an event listener for the 'loaded' event
// If not added synchronously, you will probably miss the event
// This event will not be emitted if an error occurs during loading
db.once('loaded', function () {
  console.log('Loaded the datastore!');
});

Type 4: Persistent datastore for a Node WebKit app

// For a Node WebKit app called 'nwtest'
// For example on Linux, the datafile will be ~/.config/nwtest/nestdb-data/something.db
var Datastore = require('nestdb')
  , path = require('path')
  , db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'something.db') });

Loading a datastore with an event listener

var Datastore = require('nestdb')
  , db = new Datastore({ filename: 'path/to/datafile', autoload: true });
// You can issue commands right away because of NestDB's internal queueing

// You can also synchronously add an event listener for the 'loaded' event
// If not added synchronously, you will probably miss the event
// This event will not be emitted if an error occurs during loading
db.once('loaded', function () {
  console.log('Loaded the datastore!');
});

Listening for the global "created" event

var Datastore = require('nestdb')
  , db;

Datastore.on('created', function (dbRef) {
  // dbRef === db
  console.log('Created or loaded a datastore: ' + (dbRef.filename || 'in-memory'));
});

db = new Datastore({ filename: 'path/to/datafile', autoload: true });

Loading multiple datastores

// Of course you can create multiple datastores if you need several collections.
// In this case, it is best to use autoload for all -- unlike this example!
db = {};
db.users = new Datastore('path/to/users.db');
db.robots = new Datastore('path/to/robots.db');

// You need to load each datastore (here we do it manually)
db.users.load();
db.robots.load();

Persistence

Under the hood, NestDB's persistence uses an append-only format, meaning that all updates and deletes actually result in lines added at the end of the datafile, for performance reasons. The datastore is automatically compacted (i.e. put back in the one-line-per-document format) every time you load each datastore within your application.

You can manually call the compaction function with yourDatastore.persistence.compactDatafile(cb), where callback is optional and get passed an error if any. It queues a compaction of the datafile in the executor, to be executed sequentially after all pending operations. The datastore will also fire a "compacted" event whenever a compaction process is finished.

You can also set automatic compaction at regular intervals with yourDatastore.persistence.setAutocompactionInterval(interval), interval in milliseconds (a minimum of 5 seconds is enforced), and stop automatic compaction with yourDatastore.persistence.stopAutocompaction().

Keep in mind that compaction takes a bit of time (not too much: 130ms for 50k records on a typical development machine) and no other operation can happen when it does, so most projects actually don't need to use it.

Compaction will also immediately remove any documents whose data line has become corrupted, assuming that the total percentage of all corrupted documents in that datastore still falls below the specified corruptAlertThreshold option's value.

Durability works similarly to major databases: compaction forces the OS to physically flush data to disk, while appends to the data file do not (the OS is responsible for flushing the data). That guarantees that a server crash can never cause complete data loss, while preserving performance. The worst that can happen is a crash between two syncs, causing a loss of all data between the two syncs. Usually syncs are 30 seconds appart so that's at most 30 seconds of data. This post by Antirez on Redis persistence explains this in more details, NestDB being very close to Redis AOF persistence with appendfsync option set to no.

Inserting documents

The supported native types are String, Number, Boolean, Date and null. You can also use arrays and subdocuments (objects). If a field is undefined, it will not be saved (this is different from MongoDB which transforms undefined in null, something I find counter-intuitive).

If the document does not contain an _id field, NestDB will automatically generate one for you (a 16-character alphanumerical string). The _id of a document, once set, cannot be modified.

Field names cannot begin by '$' nor contain a '.'.

Once a document is fully inserted, its datastore will also emit an "inserted" event that you can add a listener for. This event is always emitted once for each single new document, even if a bulk-insert is done.

Example 1: Inserting a single document

var doc = { hello: 'world'
               , n: 5
               , today: new Date()
               , nestdbIsAwesome: true
               , notthere: null
               , notToBeSaved: undefined  // Will not be saved
               , fruits: [ 'apple', 'orange', 'pear' ]
               , infos: { name: 'nestdb' }
               };

db.insert(doc, function (err, newDoc) {   // Callback is optional
  // newDoc is the newly inserted document, including its _id
  // newDoc has no key called notToBeSaved since its value was undefined
});

Example 2: Inserting multiple documents

You can also bulk-insert an array of documents. This operation is atomic, meaning that if one insert fails due to a unique constraint being violated, all changes are rolled back.

db.insert([{ a: 5 }, { a: 42 }], function (err, newDocs) {
  // Two documents were inserted in the datastore
  // newDocs is an array with these documents, augmented with their _id
});

// If there is a unique constraint on field 'a', this will fail
db.insert([{ a: 8 }, { a: 14 }, { a: 8 }], function (err) {
  // err is a 'uniqueViolated' error
  // The datastore was not modified
});

Example 3: Inserting a document with an event listener

db.on('inserted', function (newDoc) {
  // newDoc is the newly inserted document, including its _id
  // This event is always emitted once for each single new document, even if a bulk-insert is done.
});

db.insert({ a: 5 });
db.insert({ a: 42 });
db.insert([{ a: 8 }, { a: 14 }]);

Finding documents

Use find to look for multiple documents matching your query, or findOne to look for one specific document. You can select documents based on field equality or use comparison operators ($lt, $lte, $gt, $gte, $in, $nin, $ne). You can also use logical operators $or, $and, $not and $where. See below for the syntax.

You can use regular expressions in two ways: in basic querying in place of a string, or with the $regex operator.

You can sort and paginate results using the cursor API (see below).

You can use standard projections to restrict the fields to appear in the results (see below).

Basic querying

Basic querying means are looking for documents whose fields match the ones you specify. You can use regular expression to match strings. You can use the dot notation to navigate inside nested documents, arrays, arrays of subdocuments and to match a specific element of an array.

// Let's say our datastore contains the following collection
// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false, satellites: ['Phobos', 'Deimos'] }
// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true, humans: { genders: 2, eyes: true } }
// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }
// { _id: 'id4', planet: 'Omicron Persei 8', system: 'futurama', inhabited: true, humans: { genders: 7 } }
// { _id: 'id5', completeData: { planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] } }

// Finding all planets in the solar system
db.find({ system: 'solar' }, function (err, docs) {
  // docs is an array containing documents Mars, Earth, Jupiter
  // If no document is found, docs is equal to []
});

// Finding all planets whose name contain the substring 'ar' using a regular expression
db.find({ planet: /ar/ }, function (err, docs) {
  // docs contains Mars and Earth
});

// Finding all inhabited planets in the solar system
db.find({ system: 'solar', inhabited: true }, function (err, docs) {
  // docs is an array containing document Earth only
});

// Use the dot-notation to match fields in subdocuments
db.find({ "humans.genders": 2 }, function (err, docs) {
  // docs contains Earth
});

// Use the dot-notation to navigate arrays of subdocuments
db.find({ "completeData.planets.name": "Mars" }, function (err, docs) {
  // docs contains document 5
});

db.find({ "completeData.planets.name": "Jupiter" }, function (err, docs) {
  // docs is empty
});

db.find({ "completeData.planets.0.name": "Earth" }, function (err, docs) {
  // docs contains document 5
  // If we had tested against "Mars" docs would be empty because we are matching against a specific array element
});


// You can also deep-compare objects. Don't confuse this with dot-notation!
db.find({ humans: { genders: 2 } }, function (err, docs) {
  // docs is empty, because { genders: 2 } is not equal to { genders: 2, eyes: true }
});

// Find all documents in the collection
db.find({}, function (err, docs) {
});

// The same rules apply when you want to only find one document
db.findOne({ _id: 'id1' }, function (err, doc) {
  // doc is the document Mars
  // If no document is found, doc is null
});

Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)

The syntax is { field: { $op: value } } where $op is any comparison operator:

$lt, $lte: less than, less than or equal
$gt, $gte: greater than, greater than or equal
$in: member of. value must be an array of values
$ne, $nin: not equal, not a member of
$exists: checks whether the document posses the property field. value should be true or false
$regex: checks whether a string is matched by the regular expression. Contrary to MongoDB, the use of $options with $regex is not supported, because it doesn't give you more power than regex flags. Basic queries are more readable so only use the $regex operator when you need to use another operator with it (see example below)

// $lt, $lte, $gt and $gte work on numbers and strings
db.find({ "humans.genders": { $gt: 5 } }, function (err, docs) {
  // docs contains Omicron Persei 8, whose humans have more than 5 genders (7).
});

// When used with strings, lexicographical order is used
db.find({ planet: { $gt: 'Mercury' }}, function (err, docs) {
  // docs contains Omicron Persei 8
})

// Using $in. $nin is used in the same way
db.find({ planet: { $in: ['Earth', 'Jupiter'] }}, function (err, docs) {
  // docs contains Earth and Jupiter
});

// Using $exists
db.find({ satellites: { $exists: true } }, function (err, docs) {
  // docs contains only Mars
});

// Using $regex with another operator
db.find({ planet: { $regex: /ar/, $nin: ['Jupiter', 'Earth'] } }, function (err, docs) {
  // docs only contains Mars because Earth was excluded from the match by $nin
});

Array fields

When a field in a document is an array, NestDB first tries to see if the query value is an array to perform an exact match, then whether there is an array-specific comparison function (for now there is only $size and $elemMatch) being used. If not, the query is treated as a query on every element and there is a match if at least one element matches.

$size: match on the size of the array
$elemMatch: matches if at least one array element matches the query entirely

// Exact match
db.find({ satellites: ['Phobos', 'Deimos'] }, function (err, docs) {
  // docs contains Mars
})
db.find({ satellites: ['Deimos', 'Phobos'] }, function (err, docs) {
  // docs is empty
})

// Using an array-specific comparison function
// $elemMatch operator will provide match for a document, if an element from the array field satisfies all the conditions specified with the `$elemMatch` operator
db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: 3 } } } }, function (err, docs) {
  // docs contains documents with id 5 (completeData)
});

db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: 5 } } } }, function (err, docs) {
  // docs is empty
});

// You can use inside #elemMatch query any known document query operator
db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: { $gt: 2 } } } } }, function (err, docs) {
  // docs contains documents with id 5 (completeData)
});

// Note: you can't use nested comparison functions, e.g. { $size: { $lt: 5 } } will throw an error
db.find({ satellites: { $size: 2 } }, function (err, docs) {
  // docs contains Mars
});

db.find({ satellites: { $size: 1 } }, function (err, docs) {
  // docs is empty
});

// If a document's field is an array, matching it means matching any element of the array
db.find({ satellites: 'Phobos' }, function (err, docs) {
  // docs contains Mars. Result would have been the same if query had been { satellites: 'Deimos' }
});

// This also works for queries that use comparison operators
db.find({ satellites: { $lt: 'Amos' } }, function (err, docs) {
  // docs is empty since Phobos and Deimos are after Amos in lexicographical order
});

// This also works with the $in and $nin operator
db.find({ satellites: { $in: ['Moon', 'Deimos'] } }, function (err, docs) {
  // docs contains Mars (the Earth document is not complete!)
});

Logical operators $or, $and, $not, $where

You can combine queries using logical operators:

For $or and $and, the syntax is { $op: [query1, query2, ...] }.
For $not, the syntax is { $not: query }
For $where, the syntax is { $where: function () { /* object is "this", return a boolean */ } }

db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }] }, function (err, docs) {
  // docs contains Earth and Mars
});

db.find({ $not: { planet: 'Earth' } }, function (err, docs) {
  // docs contains Mars, Jupiter, Omicron Persei 8
});

db.find({ $where: function () { return Object.keys(this).length > 6; } }, function (err, docs) {
  // docs with more than 6 properties
});

// You can mix normal queries, comparison queries and logical operators
db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }], inhabited: true }, function (err, docs) {
  // docs contains Earth
});

Sorting and paginating

If you don't specify a callback to find, findOne or count, a Cursor object is returned. You can modify the cursor with sort, skip and limit and then execute it with exec(callback).

// Let's say the datastore contains these 4 documents
// doc1 = { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false, satellites: ['Phobos', 'Deimos'] }
// doc2 = { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true, humans: { genders: 2, eyes: true } }
// doc3 = { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }
// doc4 = { _id: 'id4', planet: 'Omicron Persei 8', system: 'futurama', inhabited: true, humans: { genders: 7 } }

// No query used means all results are returned (before the Cursor modifiers)
db.find({}).sort({ planet: 1 }).skip(1).limit(2).exec(function (err, docs) {
  // docs is [doc3, doc1]
});

// You can sort in reverse order like this
db.find({ system: 'solar' }).sort({ planet: -1 }).exec(function (err, docs) {
  // docs is [doc1, doc3, doc2]
});

// You can sort on one field, then another, and so on like this:
db.find({}).sort({ firstField: 1, secondField: -1 }) ...   // You understand how this works!

Projections

You can give find and findOne an optional second argument, projections. The syntax is the same as MongoDB: { a: 1, b: 1 } to return only the a and b fields, { a: 0, b: 0 } to omit these two fields. You cannot use both modes at the time, except for _id which is by default always returned and which you can choose to omit. You can project on nested documents.

// Same datastore as above

// Keeping only the given fields
db.find({ planet: 'Mars' }, { planet: 1, system: 1 }, function (err, docs) {
  // docs is [{ planet: 'Mars', system: 'solar', _id: 'id1' }]
});

// Keeping only the given fields but removing _id
db.find({ planet: 'Mars' }, { planet: 1, system: 1, _id: 0 }, function (err, docs) {
  // docs is [{ planet: 'Mars', system: 'solar' }]
});

// Omitting only the given fields and removing _id
db.find({ planet: 'Mars' }, { planet: 0, system: 0, _id: 0 }, function (err, docs) {
  // docs is [{ inhabited: false, satellites: ['Phobos', 'Deimos'] }]
});

// Failure: using both modes at the same time
db.find({ planet: 'Mars' }, { planet: 0, system: 1 }, function (err, docs) {
  // err is the error message, docs is undefined
});

// You can also use it in a Cursor way but this syntax is not compatible with MongoDB
db.find({ planet: 'Mars' }).projection({ planet: 1, system: 1 }).exec(function (err, docs) {
  // docs is [{ planet: 'Mars', system: 'solar', _id: 'id1' }]
});

// Project on a nested document
db.findOne({ planet: 'Earth' }).projection({ planet: 1, 'humans.genders': 1 }).exec(function (err, doc) {
  // doc is { planet: 'Earth', _id: 'id2', humans: { genders: 2 } }
});

Counting documents

You can use count to count documents. It has the same syntax as find. For example:

// Count all planets in the solar system
db.count({ system: 'solar' }, function (err, count) {
  // count equals to 3
});

// Count all documents in the datastore
db.count({}, function (err, count) {
  // count equals to 4
});

Updating documents

db.update(query, update, options, callback) will update all documents matching query according to the update rules:

query is the same kind of finding query you use with find and findOne
update specifies how the documents should be modified. It is either a new document or a set of modifiers (you cannot use both together, it doesn't make sense!)
- A new document will replace the matched docs
- The modifiers create the fields they need to modify if they don't exist, and you can apply them to subdocs. Available field modifiers are $set to change a field's value, $unset to delete a field, $inc to increment a field's value and $min/$max to change field's value, only if provided value is less/greater than current value. To work on arrays, you have $push, $pop, $addToSet, $pull, and the special $each and $slice. See examples below for the syntax.
options is an object with two possible parameters
- multi (defaults to false) which allows the modification of several documents if set to true
- upsert (defaults to false) if you want to insert a new document corresponding to the update rules if your query doesn't match anything. If your update is a simple object with no modifiers, it is the inserted document. In the other case, the query is stripped from all operator recursively, and the update is applied to it.
- returnUpdatedDocs (defaults to false, not MongoDB-compatible) if set to true and update is not an upsert, will return the array of documents matched by the find query and updated. Updated documents will be returned even if the update did not actually modify them.
callback (optional) signature: (err, numAffected, affectedDocuments, upsert). Warning: the API was changed between v1.7.4 and v1.8.0. Please refer to the change log to see the change.
- For an upsert, affectedDocuments contains the inserted document and the upsert flag is set to true.
- For a standard update with returnUpdatedDocs flag set to false, affectedDocuments is not set.
- For a standard update with returnUpdatedDocs flag set to true and multi to false, affectedDocuments is the updated document.
- For a standard update with returnUpdatedDocs flag set to true and multi to true, affectedDocuments is the array of updated documents.

Note: You cannot change a document's _id.

Once a document is fully updated, its datastore will also emit an "updated" event that you can add a listener for. This event is always emitted once for each single updated document, even if a bulk-update is done.

If a document is upserted rather than updated, the datastore will emit an "inserted" event instead.

Examples

// Let's use the same example collection as in the "finding document" part
// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false }
// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true }
// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }
// { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true }

// Replace a document by another
db.update({ planet: 'Jupiter' }, { planet: 'Pluton'}, {}, function (err, numReplaced) {
  // numReplaced = 1
  // The doc #3 has been replaced by { _id: 'id3', planet: 'Pluton' }
  // Note that the _id is kept unchanged, and the document has been replaced
  // (the 'system' and inhabited fields are not here anymore)
});

// Set an existing field's value
db.update({ system: 'solar' }, { $set: { system: 'solar system' } }, { multi: true }, function (err, numReplaced) {
  // numReplaced = 3
  // Field 'system' on Mars, Earth, Jupiter now has value 'solar system'
});

// Setting the value of a non-existing field in a subdocument by using the dot-notation
db.update({ planet: 'Mars' }, { $set: { "data.satellites": 2, "data.red": true } }, {}, function () {
  // Mars document now is { _id: 'id1', system: 'solar', inhabited: false
  //                      , data: { satellites: 2, red: true }
  //                      }
  // Not that to set fields in subdocuments, you HAVE to use dot-notation
  // Using object-notation will just replace the top-level field
  db.update({ planet: 'Mars' }, { $set: { data: { satellites: 3 } } }, {}, function () {
    // Mars document now is { _id: 'id1', system: 'solar', inhabited: false
    //                      , data: { satellites: 3 }
    //                      }
    // You lost the "data.red" field which is probably not the intended behavior
  });
});

// Deleting a field
db.update({ planet: 'Mars' }, { $unset: { planet: true } }, {}, function () {
  // Now the document for Mars doesn't contain the planet field
  // You can unset nested fields with the dot notation of course
});

// Upserting a document
db.update({ planet: 'Pluton' }, { planet: 'Pluton', inhabited: false }, { upsert: true }, function (err, numReplaced, upsert) {
  // numReplaced = 1, upsert = { _id: 'id5', planet: 'Pluton', inhabited: false }
  // A new document { _id: 'id5', planet: 'Pluton', inhabited: false } has been added to the collection
});

// If you upsert with a modifier, the upserted doc is the query modified by the modifier
// This is simpler than it sounds :)
db.update({ planet: 'Pluton' }, { $inc: { distance: 38 } }, { upsert: true }, function () {
  // A new document { _id: 'id5', planet: 'Pluton', distance: 38 } has been added to the collection  
});

// If we insert a new document { _id: 'id6', fruits: ['apple', 'orange', 'pear'] } in the collection,
// let's see how we can modify the array field atomically

// $push inserts new elements at the end of the array
db.update({ _id: 'id6' }, { $push: { fruits: 'banana' } }, {}, function () {
  // Now the fruits array is ['apple', 'orange', 'pear', 'banana']
});

// $pop removes an element from the end (if used with 1) or the front (if used with -1) of the array
db.update({ _id: 'id6' }, { $pop: { fruits: 1 } }, {}, function () {
  // Now the fruits array is ['apple', 'orange']
  // With { $pop: { fruits: -1 } }, it would have been ['orange', 'pear']
});

// $addToSet adds an element to an array only if it isn't already in it
// Equality is deep-checked (i.e. $addToSet will not insert an object in an array already containing the same object)
// Note that it doesn't check whether the array contained duplicates before or not
db.update({ _id: 'id6' }, { $addToSet: { fruits: 'apple' } }, {}, function () {
  // The fruits array didn't change
  // If we had used a fruit not in the array, e.g. 'banana', it would have been added to the array
});

// $pull removes all values matching a value or even any NestDB query from the array
db.update({ _id: 'id6' }, { $pull: { fruits: 'apple' } }, {}, function () {
  // Now the fruits array is ['orange', 'pear']
});
db.update({ _id: 'id6' }, { $pull: { fruits: $in: ['apple', 'pear'] } }, {}, function () {
  // Now the fruits array is ['orange']
});

// $each can be used to $push or $addToSet multiple values at once
// This example works the same way with $addToSet
db.update({ _id: 'id6' }, { $push: { fruits: { $each: ['banana', 'orange'] } } }, {}, function () {
  // Now the fruits array is ['apple', 'orange', 'pear', 'banana', 'orange']
});

// $slice can be used in conjunction with $push and $each to limit the size of the resulting array.
// A value of 0 will update the array to an empty array. A positive value n will keep only the n first elements
// A negative value -n will keep only the last n elements.
// If $slice is specified but not $each, $each is set to []
db.update({ _id: 'id6' }, { $push: { fruits: { $each: ['banana'], $slice: 2 } } }, {}, function () {
  // Now the fruits array is ['apple', 'orange']
});

// $min/$max to update only if provided value is less/greater than current value
// Let's say the datastore contains this document
// doc = { _id: 'id', name: 'Name', value: 5 }
db.update({ _id: 'id1' }, { $min: { value: 2 } }, {}, function () {
  // The document will be updated to { _id: 'id', name: 'Name', value: 2 }
});

db.update({ _id: 'id1' }, { $min: { value: 8 } }, {}, function () {
  // The document will not be modified
});


// Use an event listener for the "updated" event
// This event is always emitted once for each single updated document, even if a bulk-update is done.
db.on('updated', function (newDoc, oldDoc) {
  // newDoc will be: { _id: id, a: 8 }
  // oldDoc will be: { _id: id, a: 5 }
});
db.insert({ a: 5 });
db.update({ a: 5 }, { $set: { a: 8 } });

Removing documents

db.remove(query, options, callback) will remove all documents matching query according to options

query is the same as the ones used for finding and updating
options only one option for now: multi which allows the removal of multiple documents if set to true. Default is false
callback is optional, signature: err, numRemoved

Once a document is fully removed, its datastore will also emit an "removed" event that you can add a listener for. This event is always emitted once for each single removed document, even if a bulk-remove is done.

// Let's use the same example collection as in the "finding document" part
// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false }
// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true }
// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }
// { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true }

// Remove one document from the collection
// options set to {} since the default for multi is false
db.remove({ _id: 'id2' }, {}, function (err, numRemoved) {
  // numRemoved = 1
});

// Remove multiple documents
db.remove({ system: 'solar' }, { multi: true }, function (err, numRemoved) {
  // numRemoved = 3
  // All planets from the solar system were removed
});

// Removing all documents with the 'match-all' query
db.remove({}, { multi: true }, function (err, numRemoved) {
});


// Use an event listener for the "removed" event
// This event is always emitted once for each single removed document, even if a bulk-remove is done.
db.on('removed', function (oldDoc) {
  // First event emission will contain oldDoc as: { _id: id, a: 8 }
  // Second event emission will contain oldDoc as: { _id: id, a: 14 }
});
db.insert({ a: 5 });
db.insert({ a: 8 });
db.insert({ a: 14 });
db.remove({ $min: { a: 8 } }, { multi: true });

Indexing

NestDB supports indexing. It gives a very nice speed boost and can be used to enforce a unique constraint on a field. You can index any field, including fields in nested documents using the dot notation. For now, indexes are only used to speed up basic queries and queries using $in, $lt, $lte, $gt and $gte. The indexed values cannot be of type array of object.

To create an index, use datastore.ensureIndex(options, cb), where callback is optional and get passed an error if any (usually a unique constraint that was violated). ensureIndex can be called when you want, even after some data was inserted, though it's best to call it at application startup. The options are:

fieldName (required): name of the field to index. Use the dot notation to index a field in a nested document.
unique (optional, defaults to false): enforce field uniqueness. Note that a unique index will raise an error if you try to index two documents for which the field is not defined.
sparse (optional, defaults to false): don't index documents for which the field is not defined. Use this option along with "unique" if you want to accept multiple documents for which it is not defined.
expireAfterSeconds (number of seconds, optional): if set, the created index is a TTL (time to live) index, that will automatically remove documents when the system date becomes larger than the date on the indexed field plus expireAfterSeconds. Documents where the indexed field is not specified or not a Date object are ignored

Note: the _id is automatically indexed with a unique constraint, no need to call ensureIndex on it.

You can remove a previously created index with datastore.removeIndex(fieldName, cb).

If your datastore is persistent, the indexes you created are persisted in the datafile, when you load the datastore a second time they are automatically created for you. No need to remove any ensureIndex though, if it is called on a datastore that already has the index, nothing happens.

db.ensureIndex({ fieldName: 'somefield' }, function (err) {
  // If there was an error, err is not null
});

// Using a unique constraint with the index
db.ensureIndex({ fieldName: 'somefield', unique: true }, function (err) {
});

// Using a sparse unique index
db.ensureIndex({ fieldName: 'somefield', unique: true, sparse: true }, function (err) {
});


// Format of the error message when the unique constraint is not met
db.insert({ somefield: 'nestdb' }, function (err) {
  // err is null
  db.insert({ somefield: 'nestdb' }, function (err) {
    // err is { errorType: 'uniqueViolated'
    //        , key: 'name'
    //        , message: 'Unique constraint violated for key name' }
  });
});

// Remove index on field somefield
db.removeIndex('somefield', function (err) {
});

// Example of using expireAfterSeconds to remove documents 1 hour
// after their creation (db's timestampData option is true here)
db.ensureIndex({ fieldName: 'createdAt', expireAfterSeconds: 3600 }, function (err) {
});

// You can also use the option to set an expiration date like so
db.ensureIndex({ fieldName: 'expirationDate', expireAfterSeconds: 0 }, function (err) {
  // Now all documents will expire when system time reaches the date in their
  // expirationDate field
});

Note: the ensureIndex function creates the index synchronously, so it's best to use it at application startup. It's quite fast so it doesn't increase startup time much (35 ms for a collection containing 10,000 documents).

Destroying a datastore

If you ever decide you want to destroy an existing datastore, you can call destroy.

Destroying a datastore will:

Remove all documents
Reset all indexes
Delete the datafile, if persistent

Once the datastore is fully destroyed, it also emits a "destroyed" event that you can add a listener for.

Whenever any datastore is fully destroyed, the Datastore object itself emits a global "destroyed" event that you can add a listener for.

Type 1: In-memory only datastore

var Datastore = require('nestdb')
  , db = new Datastore();
// You can issue commands right away

db.destroy();

Type 2: Persistent datastore destroyed with a callback

var Datastore = require('nestdb')
  , db = new Datastore({ filename: 'path/to/datafile', autoload: true });
// You can issue commands right away because of NestDB's internal queueing

db.destroy(function (err) {
  if (err) {
    console.error('Failed to destroy datastore:', err);
  } else {
    console.log('Destroyed the datastore!');
  }
});

Type 3: Persistent datastore destroyed with an event listener

var Datastore = require('nestdb')
  , db = new Datastore({ filename: 'path/to/datafile', autoload: true });
// You can issue commands right away because of NestDB's internal queueing

// This event will not be emitted if an error occurs during destroying
db.once('destroyed', function () {
  console.log('Destroyed the datastore!');
});

db.destroy();

Listening for the global "destroyed" event

var Datastore = require('nestdb')
  , db = new Datastore({ filename: 'path/to/datafile', autoload: true });
// You can issue commands right away because of NestDB's internal queueing

Datastore.on('destroyed', function (dbRef) {
  // dbRef === db
  console.log('Destroyed a datastore: ' + (dbRef.filename || 'in-memory'));
});

db.destroy();

Extending with plugins

Extending NestDB with plugins is easy! There are two different implementation patterns available for plugins.

Regardless of which pattern you choose, the Datastore.plugin() method returns the Datastore object itself, making it easy to chain multiple plugin calls serially.

API extension object

First, you can pass in an object. Each key in the object will be added as a property to the Datastore.prototype. It intentionally does NOT check for conflicts, so be mindful when naming your properties that you may be overwriting an existing property.

Datastore.plugin({
  // Add an instance method to all `Datastore` instances
  sayHello: function () {
    console.log('Hello!');
  }
});

var db = new Datastore();
db.sayHello();  // prints "Hello!"

This will add a sayHello instance method to all datastores, which runs associated function. It will always be called in context, so that within the function, this refers to the datastore instance.

API extension function

Alternatively, instead of passing in an object to Datastore.plugin(), you can pass in a function that takes the Datastore object and performs whatever operations you want on it. You can use this to load multiple plugins, add adapters, or attach event listeners to the Datastore object.

Datastore.plugin(function (Datastore) {

  // Add a static method to the `Datastore` object
  Datastore.hello = function () {
    console.log('world');
  };

  // Add an instance method to all `Datastore` instances
  // This is basically equivalent to using the API extension object
  Datastore.prototype.goodbye = function () {
    console.log('cruel world');
  };

});

Datastore.hello();  // prints "world"

var db = new Datastore();
db.goodbye();  // prints "cruel world"

Using a custom storage engine

As mentioned in the Creating/loading a datastore section above, NestDB allows you to specify a custom storage engine for a persistent Datastore instance by passing in a custom options.storage object to the constructor.

This is useful for leveraging NestDB as a database wrapper around non-standard file systems, such as React Native or WinJS.

The required interface to implement for a custom storage engine is as follows:

`storage.init`

Purpose: Immediately before loading a datastore, ensure that either:

a datafile exists
- The default implementation also ensures that the datafile contains all the data even if there was a crash during a full file write by attempting to recover it from a temporary datafile if needed.
or a new empty datafile is created
- The default implementation also takes advantage of this to create any missing directories in the file's path.