Awesome
[Attention] This project is not maintained.
We have moved to a new product called Vald
NGTD: Serving NGT over HTTP or gRPC. 
Description
NGTD provides serving function for NGT.
NGTD supports gRPC and HTTP protocol, so you can implement applications with your favorite programming language.
You can set any labels for each vectors, and enable to search with the label.
Install
You must install NGT before installing ngtd.
The easiest way to install is the following command:
$ go get -u github.com/yahoojapan/ngtd/cmd/ngtd
Docker
You can get ngtd docker image with the following command.
$ docker pull yahoojapan/ngtd
The image include only NGTD single binary, you enable to run the docker container.
Usage
$ ngtd --help
NAME:
ngtd - NGT Daemonize
USAGE:
ngtd [global options] command [command options] [arguments...]
VERSION:
0.0.1-first
COMMANDS:
http, H serve ngtd index by http
grpc, g serve ngtd index by grpc
build, b build ngtd index
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--help, -h show help
--version, -v print the version
Serve
ngtd supports http and gRPC protocol.
HTTP
$ ngtd http --help
NAME:
ngtd http - serve ngtd index by http
USAGE:
ngtd http [command options] [arguments...]
OPTIONS:
--index value, -i value path to index (default: "/usr/share/ngtd/index")
--dimension value, -d value vector dimension size.(Must set if create new index) (default: -1)
--database-type value, -t value ngtd inner kvs type(redis, golevel, bolt or sqlite)
--database-path value, -p value ngtd inner kvs path(for golevel, bolt and sqlite) (default: "/usr/share/ngtd/db/kvs.db")
--redis-host value redis running host (default: "localhost")
--redis-port value redis running port (default: "6379")
--redis-password value redis password
--redis-database-index value, -I value list up 2 redis database indexes (default: 0, 1)
--port value, -P value listening port (default: 8200)
Request
$ curl -H 'Content-Type: application/json' -X POST http://localhost:8200/search -d '{"vector":[...], "size": 10, "epsilon": 0.01}'
$ curl -H 'Content-Type: application/json' -X POST http://localhost:8200/searchbyid -d '{"id":"<id>", "size": 10, "epsilon": 0.01}'
If you want more information, please read model.go
gRPC
$ ngtd grpc --help
NAME:
ngtd grpc - serve ngtd index by grpc
USAGE:
ngtd grpc [command options] [arguments...]
OPTIONS:
--index value, -i value path to index (default: "/usr/share/ngtd/index")
--dimension value, -d value vector dimension size.(Must set if create new index) (default: -1)
--database-type value, -t value ngtd inner kvs type(redis, golevel, bolt or sqlite)
--database-path value, -p value ngtd inner kvs path(for golevel, bolt and sqlite) (default: "/usr/share/ngtd/db/kvs.db")
--redis-host value redis running host (default: "localhost")
--redis-port value redis running port (default: "6379")
--redis-password value redis password
--redis-database-index value, -I value list up 2 redis database indexes (default: 0, 1)
--port value, -P value listening port (default: 8200)
Client
If you use language except golang, compile proto file for the language.
Go examples are in example/.
Build
Build will construct the NGT database by importing the Vector data, please check the Input Format section at the bottom of the Build chapter for the file format to import.
$ ngtd build --help
NAME:
ngtd build - build ngtd index
USAGE:
ngtd build [command options] [arguments...]
OPTIONS:
--index value, -i value path to index (default: "/usr/share/ngtd/index")
--dimension value, -d value vector dimension size.(Must set if create new index) (default: -1)
--database-type value, -t value ngtd inner kvs type(redis, golevel, bolt or sqlite)
--database-path value, -p value ngtd inner kvs path(for golevel, bolt and sqlite) (default: "/usr/share/ngtd/db/kvs.db")
--redis-host value redis running host (default: "localhost")
--redis-port value redis running port (default: "6379")
--redis-password value redis password
--redis-database-index value, -I value list up 2 redis database indexes (default: 0, 1)
--text-delimiter value, -D value delimiter for text input (default: "\t", " ")
--pool value number of CPU using NGT indexing (default: 8)
--parallel-parse value number of CPU using input parser (default: 8)
Input format
Now, we support only text format.
It enables for id to use any character without delimiter1 and for vector elements to use decimal/hex format.
If you choice hex format, you must begin the value with "0x".
<id1><delimiter1><v11><delimiter2><v12><delimiter2>...<delimiter2><v1d>\n
<id2><delimiter1><v21><delimiter2><v22><delimiter2>...<delimiter2><v2d>\n
...
<id1><delimiter1><vn1><delimiter2><vn2><delimiter2>...<delimiter2><vnd>\n
License
Copyright (C) 2018 Yahoo Japan Corporation
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Contributor License Agreement
This project requires contributors to agree to a Contributor License Agreement (CLA).
Note that only for contributions to the ngtd repository on the GitHub (https://github.com/yahoojapan/ngtd), the contributors of them shall be deemed to have agreed to the CLA without individual written agreements.
Authors
