Home

Awesome

Jupyter Quant

A dockerized Jupyter quant research environment.

We are moving!!! This repository is now maintained here -> https://github.com/quantbelt/jupyter-quant/

Highlights

Quick Start

Create a docker-compose.yml file with this content

services:
  jupyter-quant:
    image: gnzsnz/jupyter-quant:${IMAGE_VERSION}
    environment:
      APT_PROXY: ${APT_PROXY:-}
      BYODF: ${BYODF:-}
      SSH_KEYDIR: ${SSH_KEYDIR:-}
      TZ: ${QUANT_TZ:-}
    restart: unless-stopped
    ports:
      - ${LISTEN_PORT}:8888
    volumes:
      - quant_conf:/home/gordon/.config
      - quant_data:/home/gordon/.local
      - ${PWD}/Notebooks:/home/gordon/Notebooks

volumes:
  quant_conf:
  quant_data:

You can use .env-dist as your starting point.

cp .env-dist .env
# verify everything looks good
docker compose config
docker compose up

Volumes

The image is designed to work with 3 volumes:

  1. quant_data - volume for ~/.local folder. It contains caches and all python packages. This enables to add additional packages through pip.
  2. quant_conf - volume for ~/.config, all config goes here. This includes jupyter, ipython, matplotlib, etc
  3. Bind mount (but you could use a named volume) - volume for all notebooks, under ~/Notebooks.

This allows to have ephemeral containers and to keep your notebooks (3), your config (2) and your additional packages (1). Eventually you would need to update the image, in this case your notebooks (3) can move without issues, your config (2) should still work but no warranty, and your packages in quant_data could still be used but you should refresh it with a new image. Eventually, you would need to refresh (1) and less frequently (2)

Common tasks

Get running server URL

docker exec -it jupyterquant jupyter-server list
Currently running servers:
http://40798f7a604a:8888/?token=
ebf9e870d2aa0ed877590eb83b4d3bbbdfbd55467422a167 :: /home/gordon/Notebooks

or

docker logs -t jupyter-quant 2>&1 | grep '127.0.0.1:8888/lab?token='

You will need to change hostname (40798f7a604a in this case) or 127.0.0.1 by your docker host ip.

Show jupyter config

docker exec -it jupyter-quant jupyter-server --show-config

Set password

docker exec -it jupyter-quant jupyter-server password

Get command line help

docker exec -it jupyter-quant jupyter-server --help
docker exec -it jupyter-quant jupyter-lab --help

List installed packages

docker exec -it jupyter-quant pip list
# outdated packages
docker exec -it jupyter-quant pip list -o

Pass parameters to jupyter-lab

docker run -it --rm gnzsnz/jupyter-quant --core-mode
docker run -it --rm gnzsnz/jupyter-quant --show-config-json

Run a command in the container

docker run -it --rm gnzsnz/jupyter-quant bash

Build wheels outside the container

Build wheels outside the container and import wheels into the container

# make sure python version match .env-dist
docker run -it --rm -v $PWD/wheels:/wheels python:3.11 bash
pip wheel --no-cache-dir --wheel-dir /wheels numpy

This will build wheels for numpy (or any other package that you need) and save the file in $PWD/wheels. Then you can copy the wheels in your notebook mount (3 above) and install it within the container. You can even drag and drop into Jupyter.

Install your dotfiles

git clone your dotfiles to Notebook/etc/dotfiles, set environment variable BYODF=/home/gordon/Notebook/etc/dotfiles in your docker-compose.yml When the container starts up stow will create links like /home/gordon/.bashrc

Install your SSH keys

You need to define environment variable SSH_KEY_DIR which should point to a location with your keys. The suggested place is SSH_KEYDIR=/home/gordon/Notebooks/etc/ssh, make sure the director has the right permissions. Something like chmod 700 Notebooks/etc/ssh should work.

The entrypoint.sh script will create a symbolic link pointing to $SSH_KEYDIR on /home/gordon/.ssh.

Within Jupyter's terminal, you can then:

# start agent
eval $(ssh-agent)
# add keys to agent
ssh-add
# open a tunnel
ssh -fNL 4001:localhost:4001 gordon@bastion-ssh