Awesome
locus
locus
is library for Erlang/OTP and Elixir that allows you to pinpoint
the country, city or ASN of IP addresses using MaxMind
GeoIP2 and other
providers.
The databases will be loaded on-demand and, when retrieved from the network, cached on the filesystem and updated automatically.
⚠️ For instructions on how to upgrade to 2.x, check
MIGRATION.md
Usage
1. Configure your license key
Skip this step if you're not loading databases directly from MaxMind.
Get a free license key from MaxMind if you haven't one already. Once logged in, you'll find the page to generate it on the left menu, under "Manage License Keys".
Then clone the repository, run make shell
and declare your key:
application:set_env(locus, license_key, "YOUR_LICENSE_KEY").
If you're using Elixir, add locus as a dependency to your mix project:
defp deps do
[
...
{:locus, "~> 2.3"}
]
end
Then, configure your license key in config.exs
:
config :locus,
license_key: <MAXMIND_API_KEY>
2. Start the database loader
ok = locus:start_loader(country, {maxmind, "GeoLite2-Country"}).
% You can also use:
% * an HTTP(S) URL,
% * or a local path, e.g. "/usr/share/GeoIP/GeoLite2-City.mmdb"
% * or a {custom_fetcher, Module, Args} tuple, with Module
% implementing the locus_custom_fetcher behaviour.
Or, in Elixir, start the database loaders that you'll be using in application.ex
:
def start(_type, _args) do
# :locus.start_loader(:asn, {:maxmind, "GeoLite2-ASN"})
# :locus.start_loader(:country, {:maxmind, "GeoLite2-Country"})
:locus.start_loader(:city, {:maxmind, "GeoLite2-City"})
...
3. Wait for the database to load (optional)
{ok, _DatabaseVersion} = locus:await_loader(country). % or `{error, Reason}'
4. Look up IP addresses
% > locus:lookup(country, "93.184.216.34").
% > locus:lookup(country, "2606:2800:220:1:248:1893:25c8:1946").
% * '{ok, Entry}' in case of success;
% * 'not_found' if no entry was found
% * '{error, _}' if something bad happened
{ok,#{<<"continent">> =>
#{<<"code">> => <<"NA">>,
<<"geoname_id">> => 6255149,
<<"names">> =>
#{<<"de">> => <<"Nordamerika">>,
<<"en">> => <<"North America">>,
<<"es">> => <<"Norteamérica"/utf8>>,
<<"fr">> => <<"Amérique du Nord"/utf8>>,
<<"ja">> => <<"北アメリカ"/utf8>>,
<<"pt-BR">> => <<"América do Norte"/utf8>>,
<<"ru">> => <<"Северная Америка"/utf8>>,
<<"zh-CN">> => <<"北美洲"/utf8>>}},
<<"country">> =>
#{<<"geoname_id">> => 6252001,
<<"iso_code">> => <<"US">>,
<<"names">> =>
#{<<"de">> => <<"USA">>,
<<"en">> => <<"United States">>,
<<"es">> => <<"Estados Unidos">>,
<<"fr">> => <<"États-Unis"/utf8>>,
<<"ja">> => <<"アメリカ合衆国"/utf8>>,
<<"pt-BR">> => <<"Estados Unidos">>,
<<"ru">> => <<"США"/utf8>>,
<<"zh-CN">> => <<"美国"/utf8>>}},
<<"registered_country">> =>
#{<<"geoname_id">> => 6252001,
<<"iso_code">> => <<"US">>,
<<"names">> =>
#{<<"de">> => <<"USA">>,
<<"en">> => <<"United States">>,
<<"es">> => <<"Estados Unidos">>,
<<"fr">> => <<"États-Unis"/utf8>>,
<<"ja">> => <<"アメリカ合衆国"/utf8>>,
<<"pt-BR">> => <<"Estados Unidos">>,
<<"ru">> => <<"США"/utf8>>,
<<"zh-CN">> => <<"美国"/utf8>>}}}}
Or, in Elixir, call the erlang library from your Elixir application:
iex> :locus.lookup(:city, "93.184.216.34")
{:ok,
%{
"city" => %{"geoname_id" => 4945936, "names" => %{"en" => "Norwell"}},
...
}}
Documentation
- Supported File Formats
- Database Types and Loading
- Database Validation
- Remote sources: Downloading and Updating
- Remote sources: Caching
- Local sources: Loading and Updating
- Logging
- Event Subscriptions
- API Reference
- Tested Setup
- License
- Alternative Providers
- Alternative Libraries (Erlang)
- Alternative Libraries (Elixir)
Supported File Formats
- gzip-compressed tarballs (
.tar.gz
,.tgz
) - plain tarballs (
.tar
) - MMDB files (
.mmdb
) - gzip-compressed MMDB files (
.mmdb.gz
)
For tarball files, the first file to be found within it with an .mmdb
extension is the one that's chosen for loading.
The implementation of MaxMind DB
format is complete except for
the data cache container
data type.
Database Types and Loading
- The free GeoLite2 Country, City and ASN
databases were all
successfully tested; presumably
locus
can deal with any MMDB database that maps IP address prefixes to arbitrary data - The databases are loaded into memory (mostly) as is; reference
counted binaries are shared with the application callers using
persistent_term
, and the original binary search tree is used to lookup addresses. The data for each entry is decoded on the fly upon successful lookups.
Database Validation
Databases, local or remote, can have their compatibility validated
through the locus:check/1
function after they've been loaded (see
function reference.)
Alternatively, they can also be checked from the command line by use of
the locus
CLI utility:
-
Run
make cli
to build the script, namedlocus
, which will be deployed to the current directory. -
Check the database:
./locus check GeoLite2-City.mmdb # Loading database from "GeoLite2-City.mmdb"... # Database version {{2019,11,6},{11,58,0}} successfully loaded # Checking database for flaws... # Database is wholesome.
The script will exit with code 1 in case of failure, and 0 otherwise.
Warnings can produce failure through the --warnings-as-errors
flag.
Run ./locus check --help
for a description of supported options and
arguments.
Remote sources: Downloading and Updating
- The downloaded database files, when compressed, are inflated in memory
- For MaxMind and HTTP downloads, the
last-modified
response header, if present, is used to condition subsequent download attempts (usingif-modified-since
request headers) in order to save bandwidth - The downloaded databases are cached on the filesystem in order to more quickly achieve readiness on future launches of the database loader
- Database download attempts are retried upon error according to an
exponential backoff policy - quickly at first (every few seconds)
but gradually slowing down to every 15 minutes. Successful and
dismissed download attempts will be checked for update after 6
hours. Both of these behaviours can be tweaked through the
error_retries
andupdate_period
loader settings (see function reference.) - When downloading from a MaxMind edition or an HTTP URL, the remote
certificate will be authenticated against a list of known
Certification
Authorities and
connection negotiation will fail in case of an expired certificate,
mismatched hostname, self-signed certificate or unknown
certification authority. These checks can be disabled by specifying
the
insecure
loader option.
Remote sources: Caching
- Caching is a best effort; the system falls back to relying exclusively on the network if needed
- By default a caching directory named
locus_erlang
is created under the 'user_cache' basedir - A cached database is named after either:
- the MaxMind database edition name (when explicitly downloading from MaxMind), or
- the SHA256 hash of the HTTP(S) URL, or
- for
{custom_fetcher, Module, Args}
sources, a filesystem-safe version ofModule
's name concatenated with the 32-biterlang:phash2/2
value of the opaque database source as returned by the callbacks.
- Modification time of the databases is retrieved from either:
- the
last-modified
response header (when present, for MaxMind and HTTP(S) sources) - the
modified_on
metadata property for successfullocus_custom_fetcher
:fetch/1
and:conditionally_fetch/2
callbacks (for databases loaded withlocus_custom_fetcher
)
- the
- Caching can be disabled by specifying the
no_cache
option when running:start_loader
- The cache database location can be customised by providing
{database_cache_file, FilePath}
option forlocus_loader
(FilePath
must have a ".mmdb.gz" extension)
Local sources: Loading and Updating
- The loaded database files, when compressed, are inflated in memory
- The database modification timestamp is used to condition subsequent load attempts in order to lower I/O activity
- Database load attempts are retried upon error according to an
exponential backoff policy - quickly at first (every few seconds)
but gradually slowing down to every 30 seconds. Successful and
dismissed load attempts will be checked for update after 30 seconds.
Both of these behaviours can be tweaked through the
error_retries
andupdate_period
loader settings (see function reference.)
Logging
- Five logging levels are supported:
debug
,info
,warning
,error
andnone
- The chosen backend is
logger if
lager is either missing or
it hasn't
removed
logger
's default handler. - The default log level is
error
; it can be changed in the application'senv
config - To tweak the log level in runtime, use
locus_logger:set_loglevel/1
Event Subscriptions
- Any number of event subscribers can be attached to a database loader
by specifying the
{event_subscriber, Subscriber}
option when starting the database - A
Subscriber
can be either a module implementing thelocus_event_subscriber
behaviour or an arbitrarypid()
- The format and content of reported events can be consulted in detail
on the
locus_event_subscriber
module documentation; most key steps in the loader pipeline are reported (download started, download succeeded, download failed, caching succeeded, loading failed, etc.)
API Reference
The API reference can be found on HexDocs.
Tested setup
- Erlang/OTP 22 or newer
- rebar3
License
MIT License
Copyright (c) 2017-2024 Guilherme Andrade
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
locus
is an independent project and has not been authorized,
sponsored, or otherwise approved by MaxMind.
Alternative Providers
- IPinfo MMDB databases are compatible with
locus
since version 2.3.8 - DB-IP.com: lite databases seem to work but setting up auto-update for them is not practical, as there's no "latest" official URL.
Alternative Libraries (Erlang)
- egeoip: IP Geolocation module, currently supporting the MaxMind GeoLite City Database
- geodata2: Application for working with MaxMind geoip2 (.mmdb) databases
- geoip: Returns the location of an IP address; based on the ipinfodb.com web service
- geolite2data: Periodically fetches the free MaxMind GeoLite2 databases
- ip2location-erlang: Uses IP2Location geolocation database
Alternative Libraries (Elixir)
- asn: IP-to-AS-to-ASname lookup
- freegeoip: Simple wrapper for freegeoip.net HTTP API
- freegeoipx: API Client for freegeoip.net
- geoip: Lookup the geo location for a given IP address, hostname or Plug.Conn instance
- geolix: MaxMind GeoIP2 database reader/decoder
- plug_geoip2: Adds geo location to a Plug connection based upon the client IP address by using MaxMind's GeoIP2 database
- tz_world: Resolve timezones rom a location efficiently using PostGIS and Ecto