Home

Awesome

Lindera ko-dic Builder

License: MIT Join the chat at https://gitter.im/lindera-morphology/lindera

ko-dic dictionary builder for Lindera. This project fork from fulmicoton's kuromoji-rs.

Install

% cargo install lindera-ko-dic-builder

Build

The following products are required to build:

% cargo build --release

Build small binary

You can reduce the size of the dictionary by using the "compress" feature flag.
Instead, it can only be used with Lindera, which supports compression.

This repo example is this.

% cargo build --release --features compress

It also depends on liblzma to compress the dictionary. Please install the dependent packages as follows:

% sudo apt install liblzma-dev

Dictionary version

This repository contains mecab-ko-dic-2.1.1-20180720.

Building a dictionary

Building a dictionary with lindera-ko-dic command:

% KO_DIC_VERSION=2.1.1-20180720
% curl -L -O "https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-${KO_DIC_VERSION}.tar.gz"
% tar zxvf ./mecab-ko-dic-${KO_DIC_VERSION}.tar.gz
% lindera-ko-dic-builder -s ./mecab-ko-dic-${KO_DIC_VERSION} -d ./lindera-ko-dic-${KO_DIC_VERSION}

Dictionary format

Information about the dictionary format and part-of-speech tags used by mecab-ko-dic id documented in this Google Spreadsheet, linked to from mecab-ko-dic's repository readme.

Note how ko-dic has one less feature column than NAIST JDIC, and has an altogether different set of information (e.g. doesn't provide the "original form" of the word).

The tags are a slight modification of those specified by 세종 (Sejong), whatever that is. The mappings from Sejong to mecab-ko-dic's tag names are given in tab 태그 v2.0 on the above-linked spreadsheet.

The dictionary format is specified fully (in Korean) in tab 사전 형식 v2.0 of the spreadsheet. Any blank values default to *.

IndexName (Korean)Name (English)Notes
0품사 태그part-of-speech tagSee 태그 v2.0 tab on spreadsheet
1의미 부류meaning(too few examples for me to be sure)
2종성 유무presence or absenceT for true; F for false; else *
3읽기readingusually matches surface, but may differ for foreign words e.g. Chinese character words
4타입typeOne of: Inflect (활용); Compound (복합명사); or Preanalysis (기분석)
5첫번째 품사first part-of-speeche.g. given a part-of-speech tag of "VV+EM+VX+EP", would return VV
6마지막 품사last part-of-speeche.g. given a part-of-speech tag of "VV+EM+VX+EP", would return EP
7표현expression활용, 복합명사, 기분석이 어떻게 구성되는지 알려주는 필드 – Fields that tell how usage, compound nouns, and key analysis are organized

Tokenizing text using produced dictionary

You can tokenize text using produced dictionary with lindera command:

% echo "하네다공항한정토트백" | lindera -d ./lindera-ko-dic-2.1.1-20180720

하네다  NNP,인명,F,하네다,*,*,*,*
공항    NNG,장소,T,공항,*,*,*,*
한정    NNG,*,T,한정,*,*,*,*
토트백  NNG,*,T,토트백,Compound,*,*,토트/NNP/인명+백/NNG/*
EOS

For more details about lindera command, please refer to the following URL:

API reference

The API reference is available. Please see following URL: