Home

Awesome

Meteor Search

Notes

  1. This is "Spike" Code. It works but has Zero Unit Tests so is for Learning Purposes Only!
  2. Was written for the previous version of Meteor and requires some tweaking for Meteor 1.0 (which we are not using)...

WHY

People want to find content. Good search is essential.

What

Full-text search. Using Native MongoDB Commands.

Example app:

Searchr screens

How

Step 1 - Get "Real" MongoDB

By default, Meteor starts up its own instance of MongoDB, this does not have full-text indexing/search, so you need to go native.

If you don't already have a "Real" installation of MongoDB on your system, install it with HomeBrew:

brew update
brew install mongodb

If you're not using Mac ...

Startup MongoDB with textSearchEnabled=true

In a terminal/console window startup up your mongod databse with the following command:

mongod --dbpath ~/code/meteor-search/.meteor/local/db --setParameter textSearchEnabled=true

Notes:

Confirm its working by visiting: http://localhost:28017/ (in your browser)

MongoDB Running Locally

More info on enabling text search: http://docs.mongodb.org/manual/tutorial/enable-text-search/

Start Meteor with the "Real" MongoDB

Because we are using "real" Mongo we need to specifiy it when starting meteor. (This only applies while you are developing in production you would set an ENV

)

MONGO_URL="mongodb://localhost:27017/meteor" meteor

If the app starts up ok, its game on! <br /> (otherwise submit a bug to this repo and I will wil try to assist you!)

Step 2 - Get (Test) Content

Seed Content (From Twitter Streaming API)

When you boot this app it will access the Twitter Streaming API and fetch thousands of tweets for you to search through (locally). (leave it running for a few minutes and you will get 10k posts. Or a few hours if you want hundreds of thousands to stress test search)

If you want LOTS of content very quickly, change the KEYWORD to news.

If you want INSANE amounts of (noisy) data (to symulate high volume), use:

var KEYWORDS = "katie, justin, kim, beyonce, miley, Obama, 1DWorld, OMG, FML, breaking, news";

Step 3 - Index the Text (in MongoDB)

Once you have some content you need to ensure that MongoDB is indexing it.

Thankfully this is quite easy with MongoDB's ensureIndex method. In our case we are simply going to index the post's body field:

db.posts.ensureIndex( { body: "text" },{ background:true } );

RoboMongo Showing Search Index Run

There's a startup script that does this automatically for you at: server/indexes.js

Depending on how much data you have already inserted, this may take some time... I had 92k posts (tweets) in my DB when I ran it and it took less than 10 seconds!

More detail on ensureIndex:

Query to Search for a Keyword

db.posts.runCommand( "text", { search: "paypal" } )

While this works fine in RoboMongo:

RoboMongo Search Results

Meteor does not support the runCommand method:

Meteor Hates runCommand

So ...

http://stackoverflow.com/questions/17159626/implementing-mongodb-2-4s-full-text-search-in-a-meteor-app/18258688#18258688

Step 4 - Highlighting Hashtags (Clickable)

I wrote a simple regular expression to turn hashtagged keywords into links. Instead of polluting the raw data with links (and bloating our records) We finde/replace the #keywords at render time (client-side) using a Handlebars template helper method:

// place this code in your main.js or inside an Meteor.isClient block
Handlebars.registerHelper('highlight', function(text) {
  var hashtagPattern = /\s*(#\w*)/gi,
    link = "/search/",
    m, match, matches = [], t, url ='';

  // initial check for hashtag in text
  if(text.indexOf("#") !== -1) {

      // find all #keywords (that have hashtags)
      while ( (match = hashtagPattern.exec(text)) ) {
        matches.push(match[0]);
      }

      // replace any #keywords with <a href="/search/keywords">#keywords</a>
      for(var j=0; j < matches.length; j++) {
        m = matches[j].replace(/\s/g, "");
        // console.log('match',m);
        url = link+m;
        url = url.replace('#',"").toLowerCase(); // remove hashtag for lookup
        t = " <a class='hashtag' href='"+url+"'>"+m+"</a> "; // replace with
        replace = new RegExp("\\s*("+m+")", 'gi');

        text = text.replace(replace, t);
      }
    }
  return text;
});

Note: the link pattern is hard-coded /search/:keywords and method is not chainable so far from perfect! Send a pull-request if you improve on it. :-)

Notes

Setting Up Full Text Search on MongoHQ

MongoHQ does not have (full) text indexing enabled by default.

MongoHQ Shows Error When Creating Index

But they were quick to help me when I asked for it: https://twitter.com/nelsonic/statuses/451758108285489152

MongoHQ Enables Text Search

You will need to set up your indexes manually with a command <br /> (either in your Mongo Client - We use RoboMongo - or the Web Interface)

MongoHQ Showing Text Index on Posts.body

Once that is set up you are good to go.

Searching Through Your Posts

In RoboMongo (or what ever MongoDB Client of your choice) use the following command to search through your collection (on the field(s) you specified as searchable)

db.COLLECTION.runCommand( "text", { search: "KEYWORDS" } );
// e.g:
db.posts.runCommand( "text", { search: "learn" } );

RoboMongo Shows Results of runCommand

Search results returned in 0.15 seconds when I had 327k posts in the collection:

Post Count

Iron Router

This project uses Iron Router for url routing. <br /> If you are not familiar with it (you should be if you're serious about using Meteor), read:

mrt add iron-router

See the routes.js file for more detail on how I've wired this up to accept request in the form: http://yoursite.com/search/:keywords

Research

Deploying to Heroku?