Awesome
HNSW.jl
Approximate Nearest Neighbor Searches using the "Hierarchical Navigable Small World" (HNSW) algorithm as described in https://arxiv.org/abs/1603.09320 .
Highlights
- Written in Julia - no non-julian dependencies
- Supports incremental index creation
- Works with arbitrary distance functions
- Is data-agnostic - can work with data of arbitrary types given a corresponding distance function
Creating an Index
An Index in this library is a struct of type HierarchicalNSW
which can be constructed using:
hnsw = HierarchicalNSW(data; metric, M, efConstruction)
data
: This is anAbstractVector
of the data points to be used.metric = Euclidean()
: The metric to use for distance calculation. Any metric defined inDistances.jl
should work as well as any type for whichevaluate(::CustomMetric, x,y)
is implemented.M = 10
: The maximum number of links per node on a level >1. Note that value highly influences recall depending on data.M0 = 2M
: The maximum number of links on the bottom layer (=1). Defaults toM0 = 2M
.efConstruction = 100
: Maximum length of dynamic link lists during index creation. Low values may reduce recall but large values increase runtime of index creation.ef = 10
: Maximum length of dynamic link lists during search. May be changed afterwards usingset_ef!(hnsw, value)
m_L = 1/log(M)
: Prefactor for random level generation.max_elements = length(data)
: May be set to a larger value in case one wants to add elements to the structure after initial creation.
Once the HierarchicalNSW
struct is initialized the search graph can be built by calling
add_to_graph!(hnsw [, indices])
which iteratively inserts all points from data
into the graph.
Optionally one may provide indices
a subset of all the indices
in data
to partially to construct the graph.
Searching
Given an initialized HierarchicalNSW
one can search for approximate nearest
neighbors using
idxs, dists = knn_search(hnsw, query, k)
where query
may either be a single point of type eltype(data)
or a vector of such points.
A simple example:
using HNSW
dim = 10
num_elements = 10000
data = [rand(dim) for i=1:num_elements]
#Intialize HNSW struct
hnsw = HierarchicalNSW(data; efConstruction=100, M=16, ef=50)
#Add all data points into the graph
#Optionally pass a subset of the indices in data to partially construct the graph
add_to_graph!(hnsw)
# optionally with a progress notification:
# step = (num_elements) ÷ 100
# add_to_graph!(hnsw) do i
# if iszero(i % step)
# @info "Processed: $(i ÷ step)%"
# end
# end
queries = [rand(dim) for i=1:1000]
k = 10
# Find k (approximate) nearest neighbors for each of the queries
idxs, dists = knn_search(hnsw, queries, k)
Multi-Threading
A multi-threaded version of this algorithm is available.
To use it, checkout the branch multi-threaded
and start the indexing with:
add_to_graph!(hnsw; multithreading=true)
For multi-threaded searches add multithreading=true
as a keyword argument to knn_search
.