Home

Awesome

Map of American Rivers

Warning (January 2018)

This project is now nearly five years old. Huzzah! It is also slowly falling out of date. Many of the ideas in it are still relevant but vector tile technology has improved significantly since I did this work. I think the tutorial is still useful, but be aware there are more modern choices for things. For example, I'd use the Mapbox vector tiles format for the tiles now, not GeoJSON.

Also the code may have rotted a little bit. The main thing that breaks is the download URLs for NHD become invalid. It's easy to update them (typically a version number increment). Pull requests welcome.

A vector tile demonstration and tutorial

By Nelson Minar <tt><nelson@monkey.org></tt><br> May 2013<br> See the live map and the source code.

<a href="http://www.somebits.com/rivers/rivers-polymaps.html#9/38.4385/-121.1270"><img src="https://raw.github.com/NelsonMinar/vector-river-map/master/sample.jpg" alt="sample map"></a>

Many thanks to Mike Bostock, Seth Fitzsimmons, Mike Migurski, and Bobby Sudekum for feedback and help.

Introduction

This project contains everything you need from start to finish to make a vector based web map of American rivers in the contiguous 48 states. This demonstration map is neither particularly beautiful nor complex, but it is a complete example of how to build a web map using tiled vector data into a web map. The source code is open source you are encouraged to read and tinker with. There are three parts to the project: data preparation, HTTP serving of vector tiles, and clients that render maps. The components integrated in this project are:

  1. NHDPlus, the source data for river flowlines.
  2. PostGIS, a geographic database.
  3. TileStache, a vector tile GeoJSON server.
  4. Gunicorn, a Python web server container.
  5. Leaflet, Polymaps, and D3.js, three Javascript libraries for rendering maps.

It's a lot of pieces, but each one is pretty simple by itself. Combined together they form a powerful open source mapping stack for serving vector data to web browsers. You're welcome to see the map running live on my server, but the real point of this project is to show developers all the pieces necessary to build their own map using vector tiles. Read on for details of how the map is constructed and be sure to check out the source code; lots of comments and a focus on readability. There are also some very detailed development notes on my work journal.

For client authors, the vector tiles are available as a service with the URL pattern http://somebits.com:8001/rivers/{z}/{x}/{y}.json. Light use only please; the server is not provisioned for real traffic.

Quick start

About vector tiles

Vector tiles are an exciting, underutilized idea to make efficient maps. Google Maps revolutioned online cartography with "slippy maps", raster maps that are a mosaïcof PNG or JPG images. But a lot of geographic data is intrinsically vector oriented, lines and polygons. Today many map servers render vector data into raster images that are then served to clients. But serving the vector data directly to the user's browser for rendering on the client can make maps that are more flexible and more efficient. Mobile apps and proprietary services like Google Maps are starting to switch to vector maps.

Open source vector mapping is still in the early stages. There are several open source vector clients: Polymaps was an early pioneer for browser maps, MapsForge and OpenScienceMap are renderers for Android, and MapBox 2 is based on a vector tile stack. There are few open data vector services although recently OpenStreetMap has experimented with serving vector tiles. (See experimental clients like Ziggy Jonsson's, Bobby Sudekum's, and Mike Bostock's.) And currently there are only a couple of open source vector tile servers. This tutorial relies on TileStache's VecTiles provider to serve our geodata. Ceramic is an alternative.

Tiling isn't necessary for all vector data; if the full dataset is small it is reasonable to serve an entire vector geometry as a single file. For example, this tutorial maps incldues an 88kb file of US state outlines. But with 10+MB of geodata it's important to only serve visible geometry, not the entire dataset. In addition scaling tiles to the current zoom level allows simplification and down-sampling to pixel visibility. But tiling requires a lot of data preparation and server setup, hence this tutorial.

Vector tiles are ultimately quite simple. Consider this tile near near Oakland (cached copy in sample-13-1316-3169.json.txt). The URL naming system is Google's convention for raster map tiles: this tile is at z=13, x=1316, y=3169. Only instead of serving a PNG image the URL serves a GeoJSON file describing the geometry inside the tile's bounding box. This example tile has 3 features in it; one for San Lorenzo Creek, one for Sulphur Creek, and one for two other unnamed flows in the tile. Each feature contains a geometry, a name, a HUC code naming the watershed, and a Strahler number characterizing the river's significance.

One tricky thing about vector tiles is what to do about features that cross tiles. In this tutorial we clip the geometry to the tile boundary and rely on the overlapping lines being drawn to make a seamless map. It's also possible to not clip, which results in redundant data but keeps features intact. A third option can be to clip geometry and re-unify it on the client before rendering.

Server prerequisites

The following is a partial list of software you need installed on your Unix system to generate and serve these maps. (Sorry Windows users, Unix is a better choice for this kind of work.) I've tested with both MacOS and Ubuntu. On the Mac, most prerequisites are available via Homebrew; see also this guide to open source geo on the Mac. On Ubuntu most software is available via apt-get. This code requires PostGIS 2; if you are running something older than Ubuntu 14.04 you may need the UbuntuGIS PPA. See below for extra Ubuntu notes. Other Linux distributions can probably install the required software via their native package system. If the code is available on PyPI I prefer to install Python code with [pip](http://www.pip- installer.org/en/latest/) rather than rely on the Mac or Ubuntu package versions.

Extra Ubuntu 16.04 details

Not quite a complete cookbook, but close:

# Install needed software with apt and PIP
apt-get install git p7zip-full python-pip postgresql-server-dev-all python-dev libevent-dev gdal-bin postgis postgresql-client postgresql pgdbf
_create a virtualenv for python 2_
pip install psycopg2 gunicorn tilestache mapbox-vector-tile==0.5.0 requests grequests shapely --allow-external PIL --allow-unverified PIL

# Ensure postgres allows you to connect without a password
echo 'select version();' | psql -w -U nelson -h localhost postgres
# If this fails, configure postgres to let you connect via TCP to localhost without password
# One option is to specify the "trust" method on 127.0.0.1/32
edit /etc/postgresql/9.5/main/pg_hba.conf

# Optionally tune postgres performance
edit /etc/postgresql/9.5/main/postgresql.conf

Project components

This project consists of several short scripts and configuration files to glue together the software components. There is precious little programming logic here, most of it is integration.

<li>Import NHDFlowline shapefiles into a table named `nhdflowline` <li>Import PlusFlowlineVAA DBF files into a table named `plusflowlinevaa` <li>Run `processNhd.sql` to create a table named `rivers` <li>Run `mergeRivers.py` to create a table named `merged_rivers` </ol>

Cartographic decisions

Most of the work in this project is plumbing, systems programming we have to do to make the engines go. The demonstration map is deliberately quite simple and unsophisticated. Even so, it contains a few decisions requiring the map maker's art.

Most of the actual cartography is being done in Javascript, in the Leaflet and Polymaps drawing scripts. This tutorial code does very little, mostly just drawing blue lines in varying thicknesses. In addition the Leaflet version has a simple popup when rivers are clicked. With the actual vector geometry and metadata available in Javascript a lot more could be done in the presentation; highlighting rivers, interactive filtering by Strahler number, combination with other vector data sources, etc.

The map clients presented here all use the Google Mercator projection, as is standard for web maps. But because the river data is vector oriented it's possible to project it in other ways. For example, see Jason Davies' Albers rivers map.

Some cartographic decisions are made on the server side. The TileStache VecTiles configuration contains an array of queries that return results at different zoom levels. At high zoom levels (say z=4) we only return rivers which are relatively big, those with a Strahler number of 6 or higher. At finer grained zoom levels we return more and smaller rivers. This per-zoom filtering both limits the bandwidth used on large scale maps and prevents the display from being overcluttered. Rendering zillions of tiny streams can be quite beautiful, but also resource intensive.

VecTiles also simplifies the geometry, serving only the precision needed at the zoom level. You can see this in action if you watch it re-render as you navigate; rivers will start to grow more bends and detail as you zoom in. TileStache does that for us automatically.

Project ideas

The map provided here is a simple tutorial demonstration. To make this a better map, some possible directions:

Conclusion

The vector river map lays out all the components required to make an open source vector map, from downloading the data to preparing it in a database to serving tiles on the Web to rendering those tiles in the browser. If this tutorial was helpful to you or you have any suggestions or questions, please feel free to email the author at <tt>nelson@monkey.org</tt>. I'm looking forward to seeing what others are inspired to do!