Awesome

Ruby port of TinySegmenter.js for tokenizing Japanese text. Ruby 1.9 or higher required.

Install

gem install tiny_segmenter or add tiny_segmenter to your Gemfile

Usage

ts = TinySegmenter.new
ts.segment("今晩は！良い天気ですね。")
# => ["今晩", "は", "！", "良い", "天気", "です", "ね", "。"]
ts.segment("今晩は！良い天気ですね。", ignore_punctuation: true)
# => ["今晩", "は", "良い", "天気", "です", "ね"]

Input text should be UTF-8 encoded.

How it works

The Naive Bayes model was trained using the RWCP corpus and optimized using L1-norm regularization (e.g. this). The resultant model is quite compact, yet (according to the author) has about a 95% accuracy rate.

License

BSD - see http://chasen.org/~taku/software/TinySegmenter/LICENCE.txt