Awesome
medea
A persistent key-value storage library.
- Can be embedded in Node.js applications.
- Support for
get
,put
, andremove
operations. - Influenced by Basho's Bitcask key-value store.
- Values can be any string or
Buffer
. - Supports atomic batch writes.
- Allows snapshots for consistent views of data.
Contents
- Example
- Install
- Usage
- db.open([directory], callback)
- db.get(key, [snapshot], callback)
- db.put(key, value, callback)
- db.remove(key, callback)
- db.createSnapshot()
- db.createBatch()
- db.write(batch, callback)
- db.sync(callback)
- db.close(callback)
- db.listKeys(callback)
- db.mapReduce(options, callback)
- db.compact(callback)
- db.destroy(callback) / medea.destroy(director, callback)
- How It Works
- Limitations
- License
Example
var medea = require('medea');
var db = medea();
db.open(function(err) {
db.put('hello', 'world!', function(err) {
db.get('hello', function(err, val) {
console.log(val.toString());
db.close();
});
});
});
Install
$ npm install medea
Usage
db.open([directory], callback)
Opens a Medea key-value store.
directory
: Optional. Defaults to medea
in the current directory.
callback
: Takes one error parameter.
db.get(key, [snapshot], callback)
Returns the value associated with the given key.
key
: identifier to retrieve the value
snapshot
: a snapshot on which to query. optional.
callback
: has the signature function (err, value)
where value
is the returned value.
db.put(key, value, callback)
Stores a value in Medea.
key
: identifier
value
: value associated with the key
callback
: function that takes an error parameter
db.remove(key, callback)
Removes an entry from Medea.
key
: identifier for the item to remove
callback
: Takes one error parameter.
db.createSnapshot()
Creates a transient snapshot of the data. Works with db.get(key, [snapshot], callback).
Returns a Snapshot object.
Example:
db.put('hello', 'world', function(err) {
var snapshot = db.createSnapshot();
db.put('hello', 'void', function(err) {
db.get('hello', snapshot, function(err, val) {
assert.equal(val.toString, 'world');
});
});
});
db.createBatch()
Creates a batch object for atomic writes. Works with db.write(batch, callback).
Returns a WriteBatch object.
db.write(batch, callback)
Commits put
and remove
operations atomically.
batch
: a WriteBatch object created with db.createBatch().
callback
: Takes one error parameter.
Example:
var batch = db.createBatch();
batch.put('key1', 'value1');
batch.remove('key2');
db.write(batch, function(err) {
console.log('batch written atomically');
});
db.sync(callback)
Performs an fsync operation on the active data and hint files.
db.close(callback)
Closes Medea.
db.listKeys(callback)
Returns an array of keys from the key-value store.
callback
: A function that takes two parameters: an error and the array of keys.
db.mapReduce(options, callback)
Experimental.
Ad-hoc map-reduce queries over the key-value pairs. The query results are not indexed. A more robust map-reduce implementation will be provided in the near future.
See examples/map_reduce.js
for an example. <!--_-->
db.compact(callback)
Runs a compaction process on the database files. Reduces size of data on disk. Deletes old log entries.
callback
: A function that takes an error parameter.
db.destroy(callback) / medea.destroy(director, callback)
Destroys the database. If the folder only includes medea-files, the folder is removed also otherwise only the medea-files are removed and the folder is kept.
How it Works
All Medea key-value stores reside on the file system in a directory. The directory is made up of data files, hint files, and lock files. All keys are held in an in-memory keydir for fast lookup. More explanation is below.
Keydir
The keydir resides in memory. It contains a key, an associated data file ID, a value size, and a file offset. When a get
is requested, a lookup happens in the keydir. The keydir entry allows one disk seek for each get
operation. When a put
or remove
occurs, the value is updated in the keydir after the update occurs on-disk. The key set must fit in memory.
Data Files
Format: {seq}.medea.data
where {seq}
is an incrementing number.
A new data file is created each time a Medea store is opened or after an active data file reaches a configured size limit. Data files are opened in an append-only mode and act as a log. For every put
and remove
a new entry is inserted into the currently active data file. Each entry contains a timestamp, the key size, the value size, the key, and the value itself. In addition, a CRC32 checksum is calculated, which can be used to test the integrity of the entry.
When existing keys are updated with new values, new entries are appended to the end of the data file. Note: When values are removed, the entry to the data file looks just like a put
, but with a special tombstone value.
Hint Files
Format: {seq}.medea.hint
where {seq}
is an incrementing number.
Each data file has an associated hint file. Entries in a hint file contain a timestamp, a key size, a value size, an offset, and the key itself. The offset points to the latest value entry for a particular key. Hint files are used to build a new keydir when a Medea store is opened. This is much quicker than reading through the data files.
Lock files
Format: medea.write.lock
Contains a process ID and the active file path.
Limitations
Currently, multiple processes cannot access the same data directory. Run one process per directory. A workaround has been developed for servers using Node's cluster
module. Check out medea-clusterify to see how it works!
Repeated use leads to fragmentation and empty files. The compaction process needs to be run for cleanup. See: db.compact(callback).
Medea's design places all keys in memory. Therefore, the entire key set must fit in memory. All values are stored on disk.
License
MIT
Copyright 2013 Apigee Corporation and Contributors