Awesome

cl-mongo

Intro

mongo is a scalable, high-performance, open source, schema-free, document-oriented database..

This is a common lisp interface that allows you to work with a mongo document database. This is not a driver (yet) as specified in the mongo documents; There is still some functionality I need to implement.

Except for gridfs most features have been implemented.

cl-mongo provides the ability to insert, update and delete documents, indexing, searching using regexs etc. In addition it supports javascript in the repl (using a commonly available lisp to javascript compiler). This allows you to do use mongodb's map-reduce from the lisp repl.

I developed this primarily using sbcl and to some extend clozure common lisp on the linux and mac osx platforms.

This should also work on clisp and allegro common lisp and windows.

multi-threading

The do-query function uses multithreading to process the results of a batch as a query is in progress. This obviously requires that multi-threading is supported. I've found that as of now (2010-10-23) sbcl on mac osx doesn't support multithreading out of the box. So you'd need to compile a new sbcl with multithreading enabled, which is easy to do.

Version

Version 0.9.0

I've added regression testing for all features currently supported by cl-mongo.

Installation

Use asdf to install cl-mongo. cl-mongo depends on many other lisp libraries (which in turn have their own dependecies). asdf is not a dependency manager, so you would need to install all dependencies as they come up as unavailable when you try to install cl-mongo.

On the other hand, cl-mongo is included in quicklisp, which should make life easier.

Testing

The cl-mongo-test package contains regression tests for all the features currently supported. cl-mongo-test is asdf-installable, but does not export it's tests. To run the quick test, do this : (use-package :cl-mongo-test) (quick-test)

Quick test will connected to a locally running mongodb instance. It uses the "foo" collection in the "test" database.

A sample session

This connects to the test database on the local mongo server listening on its default port.

(use-package :cl-mongo)
(db.use "test")

Insert a key-value pair as the first document into collection "foo".

(db.insert "foo" (kv "document" "one") )

Pretty-print the documents returned by the find command. iter will ensure that the cursor is fully iterated over.

(pp (iter (db.find "foo" :all)))`

 {
    "_id" -> objectid(4B5CF28970DFF196A75FE1F0)
    "document"  ->  one
 }

Create a document. A document is collection of key-value pairs and a unique identifier called "_id".

(defvar *DOC* (make-document))

Add various elements to the document.

   (add-element "tag" "key" *DOC*)`


   {  DOCUMENT 

   {
   tag  -> key 
   }
   }


(add-element "array" (list 1 2 3 "hello") *DOC*)

{  DOCUMENT 

{
tag  -> key 
	 1 
	 2 
	 3 
	 hello 
 array  -> NIL 
 }
 }

Insert document into the database.

(db.insert "foo" *DOC*)

Print the current contents.

   (pp (iter (db.find "foo" 'all)))

   {
   "_id" -> objectid(4B5CF28970DFF196A75FE1F0)
   "document"  ->  one
   }

   {
   "_id" -> objectid(8B508D5CBB5D451D961F046D)
   "array"  -> [ 1, 2, 3, hello,]
   "tag"  ->  key
   }

Bind variable *DOC* to the second document returned by the find command, add an other element and save back to the collection.

  (defvar *DOC* (cadr (docs (db.find "foo" :all))))`
  (add-element "tags" (list 'good 'bad 'ugly) *DOC*)
  (db.save "foo" *DOC*)
  (pp (db.find "foo" :all))

  {
"_id" -> objectid(4B5CF28970DFF196A75FE1F0)
  	"document"  ->  one
  }

  {
    "_id" -> objectid(8B508D5CBB5D451D961F046D)
  	"tags"  -> [ GOOD, BAD, UGLY,]
 	"tag"  ->  key
 	"array"  -> [ 1, 2, 3, hello,]
  }

Check the status of the server.

  (db.use "admin")
  (nd (db.run-command :serverstatus))

    "ok"  ->  1.0d0
  "mem"  -> 
    "mapped"  ->  80
    "virtual"  ->  177
        "resident"  ->  3
    "globalLock"  -> 
      "ratio"  ->  4.644753171959394d-5
      "lockTime"  ->  6493354.0d0
    "totalTime"  ->  1.39799764586d11
    "uptime"  ->  139799.0d0

$ syntax

I've added various $.. macros which allow for a more declarative programming style. In stead of doing something like :

(db.find "foo" (kv "k" (kv "$lte" 3)))

you can use :

(db.find "foo" ($<= "k" 3))

To declare a unique ascending index on "k" in collection "foo" , you can say :

($index "foo" :unique :asc "k")

What's missing

At least the following is missing :

• Request id/ Response id are left 0 in the header.
• GridFS
• ......

API documentation

I use Edi Weitz's documentation template to generate an api description based on embedded comments.

News

2010-10-23 I've improved the read performance several orders of magnitude. This also reduces the memory foot pront quite a bit.

In addition I've added a multi-threaded reader called do-query.

The code comes with a <a href="http://www.opensource.org/licenses/bsd-license.php">MIT-style license</a> so you can basically do with it whatever you want.

<br> <br><h3><a class=none name="contents">Contents</a></h3>

<br> <br><h3><a class=none name="download">Download</a></h3>

CL-MONGO together with this documentation can be downloaded from <a href="http://github.com/fons/cl-mongo">http://github.com/fons/cl-mongo</a>. The current version is 0.1.0.

<br> <br><h3><a class=none name="dictionary">The CL-MONGO dictionary</a></h3>

Run map reduce on the mongo server. map and reduce are either the names of the javascript functions, created with defjs or defsrvjs or are function definitions in javascript. The keywords refer to option available for map reduce in mongo. This returns a result summary document. When using :keeptemp t without specificing :out the collection is mr.<collection>

database opened by the default connection

host for the default connection.

port for the default connection.

root of the repository; used for documentation generation

add element with key and value to a document

Show the current database.

Generate a time stamp the mongo/bson protocol understands.

Add a user to the database.

authenticate a user with a password

Show all the collections in the current database.

Count all the collections satifying the criterion set by the selector. :all can be used to return a count of all the documents in the collection.

create a collection

Delete a document from a collection. The document field is used to identify the document to be deleted.
You can enter a list of documents. In that the server will be contacted to delete each one of these. It may be more efficient to run a delete script on the server side.

Return all the distinct values of this key in the collection

Create an index specified by the keys in a collection

run javascript code server side

Find documents in the collection using the selector specified by kv.
Methods take two keywords. ':limit' sets the maximum number of documents returned. The default is 1. ':skip' sets the number of documents to skip in this query. It's default is 0. Since the default value of the limit is one, db.find by default is the equivalant of findOne in the mongo documentation.

Return all indexes in the database.

Insert a document in a collection. A document is typically generated by (make-document), but it can also be a hash table, a key-value pair or kv list (see the kv functions).

next document iteration

Executes the next call on the iterator identified by cursor-id.

Run a database command on the server. See the mongo documentation for a list of commands.
For most commands you can just uses the key-value shown in the mongo documentation.

Save a document to the collection. If the document has a unique _id value (i.e. if it's generated by (make-document) ) it will be 'upserted' (that is: it will be inserted if the document doesn't exist). If the document a hash table or a kv set, it will be inserted.
In other words this a a helper-function build around db.insert and db.update.

sort macro : Takes the same arguments and keywords as db.find but converts the query so it works as a sort. use the :field keyword to select the field to sort on. Set :asc to nil to reverse the sort order

Stop iterating and clean up the iterator on the server by making a server call.

In a collection update the document(s) identified by the selector statement.
This method has two keywords. ':upsert' : If t insert the document if the document cannot be found in the collection. ':multi' : Update all documents identified by the selector.

Use a database on the mongo server. Opens a connection if one isn't already established. (db.use -) can be used to go to a previosuly visited database, similar to cd -.

Define client side javascript. Works like defun; body is in lisp, with parenscript additions, like return. So (defjs hello (x y) (return (+ x y))) defines an adder. macro creates a lisp function which sends the javascript function over to the mongo server to be evaluated. Result is processed and returned to the reader. This will execute 10 times on the server : (mapcar (lambda (x) (hello 10 x)) (list 1 2 3 4 5 6 7 8 9 10))

Creates a function which stores and executes javascript on the server. The first time the function is called the javascript function is stored on the server. Subsequent calls will call out to the server. Works like defun; the function body is defined in lisp, with parenscript additions. Since the body of the function already resides on the server this should have less impact on the network. Use :install t to reinstall.

Performs a multi-threaded query on a mongo database. coll is the collection name. The reduce-fn keyword is used to specify a function which will be called for each member of a batch of data returned from mongodb.
The reduce-fn function is executed while the query for the next batch is in progress. The default for reduce-fn is the identity function. The reduction keyword is used to specify a function which is executed when the database queries have finished. It's default implementation is to return a hash table, with the mongodb id as the hash key. The query, mongo, limit and selector keywords are used in the same way as for db.find.

return the unique document id

Stop the iterator (if any) and return the list of documents returned by the query. Typical ue would be in conjunction with db.find like so (docs (iter (db.find 'foo' 'll)))

document Document class. A document consists of key/value pairs stored in a internal hash table plus an internally generated unique id.
Accessors are : 'elements' which returns the internal hash table; '_id' which returns the unique id and '_local_id' which if true means that the document was generated by the client (as opposed to having been read from the server).

This function generates a README.md file with the latest api description. The :path keyword specifies the location. It expects a sub-directory <path>/doc. Api documentation is generated on the fly, by extracting comments from the classes, generics and fuctions exported in the packages file. The resulting file is <path>/doc/index.html. <path>/README.md is generated by
appending the api documentation to <path>/doc/readme-base.md. :path or REPO-ROOT are typically set to the root of the repository.

Get an element identified by key from the document.

Convert a hash-table to a document.

Allows server based javascripts the be installed without being run.

Exhaustively iterate through a query. The maximum number of responses per query can be specified using the max-per-call keyword.

Return the body of the javascript function; otherwise nill.

This a helper function for key-value pairs and sets of key-value pairs.
In a key-value pair like (kv key value) the key has to be a string and the value something which is serializable. key-value pairs can be combined using kv as well : (kv (kv key1 val1) (kv key2 val2)). This combination of key-value pairs is equivalent to a document without a unique id. The server will assign a unique is if a list of key-value pairs is saved.

Constructor. key ':oid' is a user supplied unique id. An internal id will be generated if none is supplied.

Encapsulates the connection to the mongo database. Each connection is a added to a global registry.

This method returns the connection referred to by the name identifier from the connection registry. The connection name is unique. If no connection with that name exists, a new connection with the supplied or default host, port and db parameters will be created. The default host is localhost; the default port is 27017; the default db is admin.

Close the connection to the mongo database. The name should uniquely identify the connection to close. This is either a mongo object or the name the object is bound to in the connection registry. To close all open connections use the special symbol 'all

Return a conection registered by this name or nil..

Show all registered connections and their session id

Swap the names of the left and right connections. Typical use would be (mongo-swap :default :alt). After the function call :default will refer to the connection previously referred to as :alt. A connection named :default is returned by (mongo) and is the default used in the api. The connections are returned in the order they were passed in (but with the names swapped between them). To re-open a connection you can say (mongo-close (mongo-swap :default (mongo :host <newhost> :portid <portid> :name :temp))) and a new default connection is registered.

remove the temporary collections created by map-reduce

remove the all collections created by map-reduce, temporary as well as permanent

show the contents of the results collection map-reduce created

Pretty-print for non-document responses, like the response to a database command.

Return the current date and time in bson format.

Show the database set by the (db.use -) command

Pretty-print the results returned from a query. To be used on the repl. This will format each server reply as if it were a document. This is obviously ok in mosty cases. See nd for an alternative.

return value bound to retval in a return document. Used for db.count, db.distinct, functions etc..

Delete all the documents returned by a query. This is not an efficient way of deleting documents as it invloves multiple trips to the server. Mongo allows for execution of java-script on the server side, which provides an alternative. Typical use would be (rm (iter (db.find 'foo' (kv 'key' 1)))), which deletes all documents in foo, with field key equal to 1.

Remove element identified by key from a document

Print a list of things. Things can be users, databases, collections in the current database, the profile and more. Things is a keyword so (show :users) will show all users.

Set the time zone appropriate for the current environment.

Creates a connection to a mongodb, makes it the default connection and evaluates the body form. args uses the same keyword set as mongo (:db :localhost :port) args is passed on to make-mongo when the connection is created.

<br> <br><h3><a class=none name="ack">Acknowledgements</a></h3>