Awesome

About

Elastic search was recently used to index the Firefox4 twitter stream and make it searchable. It's based on Lucene and has a simple JSON based interface that you can use to store objects and search through them (for instance even with CURL).

This also makes it easy to have your search indexes be updated in realtime whenever your CakePHP models change data. Cause basically all we'd have to do is do a Curl PUT, DELETE, etc to also make the change in Elastisearch with every afterSave and afterDelete.

This plugin provides

• a behavior to automatically update your indexes
• a shell task to do full index fills
• a generic search component that you can attach to your AppController and will intercept search actions on enabled models. Will return results in JSON format for easy AJAX integration.

Installation

You'll need the PHP curl libraries installed.

Server

On Debian/Ubuntu

CakePHP Plugin

As a fake submodule

cd ${YOURAPP}/Plugin
git clone git://github.com/kvz/cakephp-elasticsearch-plugin.git Elasticsearch

As a real submodule

cd ${REPO_ROOT}
git submodule add git://github.com/kvz/cakephp-elasticsearch-plugin.git ${YOURAPP}/Plugin/Elasticsearch

Using composer

"require": {
	"kvz/elasticsearch": "dev-master"
}

Integration

Database

Config/database.php

<?php
class DATABASE_CONFIG {
	public $elastic = array(
		'host' => '127.0.0.1',
		'port' => '9200',
	);
	// ... etc
?>

Model

Models/Ticket.php (minimal example)

<?php
public $actsAs = array(
	'Elasticsearch.Searchable' => array(

	),
	// ... etc
);
?>

Models/Ticket.php (with raw sql for huge datasets)

<?php
public $actsAs = array(
	'Elasticsearch.Searchable' => array(
		'index_chunksize' => 1000, // per row, not per parent object anymore..
		'index_find_params' => '
			SELECT
				`tickets`.`cc` AS \'Ticket/cc\',
				`tickets`.`id` AS \'Ticket/id\',
				`tickets`.`subject` AS \'Ticket/subject\',
				`tickets`.`from` AS \'Ticket/from\',
				`tickets`.`created` AS \'Ticket/created\',
				`customers`.`customer_id` AS \'Customer/customer_id\',
				`customers`.`name` AS \'Customer/name\',
				`ticket_responses`.`id` AS \'TicketResponse/{n}/id\',
				`ticket_responses`.`from` AS \'TicketResponse/{n}/from\',
				`ticket_responses`.`created` AS \'TicketResponse/{n}/created\'
			FROM `tickets`
			LEFT JOIN `ticket_responses` ON `ticket_responses`.`ticket_id` = `tickets.id`
			LEFT JOIN `customers` ON `customers`.`customer_id` = `tickets`.`customer_id`
			WHERE 1=1
				{single_placeholder}
			{offset_limit_placeholder}
		',
	),
	// ... etc
);
?>

Models/Ticket.php (full example)

<?php
public $actsAs = array(
	'Elasticsearch.Searchable' => array(
		'debug_traces' => false,
		'searcher_enabled' => false,
		'searcher_action' => 'searcher',
		'searcher_param' => 'q',
		'searcher_serializer' => 'json_encode',
		'fake_fields' => array(
			'_label' => array('Product/description', 'BasketItem/description'),
		),
		'index_name' => 'main',
		'index_chunksize' => 10000,
		'index_find_params' => array(
			'limit' => 1,
			'fields' => array(
				// It's important you name your fields.
				'subject',
				'from',
			),
			'contain' => array(
				'Customer' => array(
					// It's important you name your fields.
					'fields' => array(
						'id',
						'name',
					),
				),
				'TicketResponse' => array(
					// It's important you name your fields.
					'fields' => array(
						'id',
						'content',
						'created',
					),
				),
				'TicketObjectLink' => array(
					// It's important you name your fields.
					'fields' => array(
						'foreign_model',
						'foreign_id',
					),
				),
				'TicketPriority' => array(
					// It's important you name your fields.
					'fields' => array(
						'code',
						'from',
					),
				),
				'TicketQueue' => array(
					// It's important you name your fields.
					'fields' => array(
						'name',
					),
				),
			),
			'order' => array(
				'Ticket.id' => 'DESC',
			),
		),
		'highlight' => array(
			'pre_tags' => array('<em class="highlight">'),
			'post_tags' => array('</em>'),
			'fields' => array(
				'_all' => array(
					'fragment_size' => 200,
					'number_of_fragments' => 1,
				),
			),
		),
		'realtime_update' => false,
		'error_handler' => 'php',
		'static_url_generator' => array('{model}', 'url'),
		'enforce' => array(
			'Customer/id' => 123,
			// or a callback: '#Customer/id' => array('LiveUser', 'id'),
		),
		'highlight_excludes' => array(
			// if you're always restricting results by customer, that
			// query should probably not be part of your highlight
			// instead of dumping _all and going over all fields except Customer/id,
			// you can also exclude it:
			'Customer/id',
		),
	),
);
?>

Controller

To automatically enable a /<controller>/searcher url on all models that have elastic search enabled, use:

Controller/AppController.php

<?php
public $components = array(
	'Elasticsearch.Searcher',
	// ... etc
);
?>

This component will only actually fire when the Controller->modelClass has the searchable behavior attached.

I chose for this method (vs a dedicated SearchesController) so ACLing is easier. e.g. You may already have an ACL for /tickets/*, so /tickets/search will automatically be restricted the same way.

Generic search

If you want to search on all models, you could make a dedicated search controller and instruct to search on everything like so:

<?php
class SearchersController extends AppController {
	public $components = array(
		'Elasticsearch.Searcher' => array(
			'model' => '_all',
			'leading_model' => 'Ticket',
		),
		// ... etc
	);

	public function searcher () {
		$this->Searcher->searchAction($this->RequestHandler->isAjax());
	}
}
?>

One known limitation is that the Elasticsearch plugin will only look at the first configured Model for configuration parameters like searcher_param and searcher_action.

Try it

From your shell:

# Fill all indexes
./cake Elasticsearch.indexer fill _all

# Fill index with tickets
./cake Elasticsearch.indexer fill Ticket

# Try a ticket search from commandline
./cake Elasticsearch.indexer search Ticket Hello

From your browser

http://www.example.com/tickets/searcher/q:*kevin*

jQuery integration

Let's look at an integration example that uses jQuery UI's autocomplete.

Assuming you have included that library, and have an input field with attributes id="main-search" and target="/tickets/searcher/q:*{query}*":

// Main-search
$(document).ready(function () {
	$("#main-search").autocomplete({
		source: function(request, response) {
			$.getJSON($("#main-search").attr('target').replace('{query}', request.term), null, response);
		},
		delay: 100,
		select: function(event, ui) {
			var id = 0;
			if ((id = ui.item.id)) {
				location.href = ui.item.url;
				alert('Selected: #' +  id + ': ' + ui.item.url);
			}
			return false;
		}
	}).data( "autocomplete" )._renderItem = function( ul, item ) {
		return $("<li></li>")
			.data("item.autocomplete", item)
			.append("<a href='" + item.url + "'>" + item.html + "<br>" + item.descr + "</a>")
			.appendTo(ul);
	};
});

Note

• There also is an unmaintained legacy cakephp 1.3 branch

Useful commands

# Get Status
curl -XGET 'http://127.0.0.1:9200/_all/_status?pretty=true'

# Dangerous: Delete an entire index
curl -XDELETE 'http://127.0.0.1:9200/main'

# Dangerous: Delete an entire type
curl -XDELETE 'http://127.0.0.1:9200/main/ticket'

# Get all tickets
curl -XGET http://127.0.0.1:9200/main/ticket/_search -d '{
	"query" : {
		"field" : {
			"_all" : "**"
		}
	}
}'

# Get everything
curl -XGET http://127.0.0.1:9200/main/_search?pretty=true -d '{
	"query" : {
		"field" : {
			"_all" : "**"
		}
	},
	"size" : 1000000
}'

# Dangerous: Delete an entire type
curl -XDELETE 'http://127.0.0.1:9200/main/ticket'

# Refresh index
curl -XPOST 'http://127.0.0.1:9200/main/_refresh'