Home

Awesome

batchspawner for Jupyterhub

Latest PyPI version Latest conda-forge version GitHub Workflow Status - Test Test coverage of code Issue tracking - GitHub Help forum - Discourse Contribute

This is a custom spawner for Jupyterhub that is designed for installations on clusters using batch scheduling software.

This began as a generalization of mkgilbert's batchspawner which in turn was inspired by Andrea Zonca's blog post where he explains his implementation for a spawner that uses SSH and Torque. His github repo is found here.

This package formerly included WrapSpawner and ProfilesSpawner, which provide mechanisms for runtime configuration of spawners. These have been split out and moved to the wrapspawner package.

Installation

  1. from root directory of this repo (where setup.py is), run pip install -e .

    If you don't actually need an editable version, you can simply run pip install batchspawner

  2. add lines in jupyterhub_config.py for the spawner you intend to use, e.g.

       c = get_config()
       c.JupyterHub.spawner_class = 'batchspawner.TorqueSpawner'
       import batchspawner    # Even though not used, needed to register batchspawner interface
    
  3. Depending on the spawner, additional configuration will likely be needed.

Batch Spawners

For information on the specific spawners, see SPAWNERS.md.

Overview

This file contains an abstraction layer for batch job queueing systems (BatchSpawnerBase), and implements Jupyterhub spawners for Torque, Moab, SLURM, SGE, HTCondor, LSF, and eventually others. Common attributes of batch submission / resource manager environments will include notions of:

BatchSpawnerBase provides several general mechanisms:

Example

Every effort has been made to accommodate highly diverse systems through configuration only. This example consists of the (lightly edited) configuration used by the author to run Jupyter notebooks on an academic supercomputer cluster.

# Select the Torque backend and increase the timeout since batch jobs may take time to start
import batchspawner
c.JupyterHub.spawner_class = 'batchspawner.TorqueSpawner'
c.Spawner.http_timeout = 120

#------------------------------------------------------------------------------
# BatchSpawnerBase configuration
#    These are simply setting parameters used in the job script template below
#------------------------------------------------------------------------------
c.BatchSpawnerBase.req_nprocs = '2'
c.BatchSpawnerBase.req_queue = 'mesabi'
c.BatchSpawnerBase.req_host = 'mesabi.xyz.edu'
c.BatchSpawnerBase.req_runtime = '12:00:00'
c.BatchSpawnerBase.req_memory = '4gb'
#------------------------------------------------------------------------------
# TorqueSpawner configuration
#    The script below is nearly identical to the default template, but we needed
#    to add a line for our local environment. For most sites the default templates
#    should be a good starting point.
#------------------------------------------------------------------------------
c.TorqueSpawner.batch_script = '''#!/bin/sh
#PBS -q {queue}@{host}
#PBS -l walltime={runtime}
#PBS -l nodes=1:ppn={nprocs}
#PBS -l mem={memory}
#PBS -N jupyterhub-singleuser
#PBS -v {keepvars}
module load python3
{cmd}
'''
# For our site we need to munge the execution hostname returned by qstat
c.TorqueSpawner.state_exechost_exp = r'int-\1.mesabi.xyz.edu'

Security

Unless otherwise stated for a specific spawner, assume that spawners do evaluate shell environment for users and thus the security requirements of JupyterHub security for untrusted users are not fulfilled because some (most?) spawners do start a user shell which will execute arbitrary user environment configuration (.profile, .bashrc and the like) unless users do not have access to their own cluster user account. This is something which we are working on.

Provide different configurations of BatchSpawner

Overview

ProfilesSpawner, available as part of the wrapspawner package, allows the Jupyterhub administrator to define a set of different spawning configurations, both different spawners and different configurations of the same spawner. The user is then presented a dropdown menu for choosing the most suitable configuration for their needs.

This method provides an easy and safe way to provide different configurations of BatchSpawner to the users, see an example below.

Example

The following is based on the author's configuration (at the same site as the example above) showing how to give users access to multiple job configurations on the batch scheduled clusters, as well as an option to run a local notebook directly on the jupyterhub server.

# Same initial setup as the previous example
import batchspawner
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.http_timeout = 120
#------------------------------------------------------------------------------
# BatchSpawnerBase configuration
#   Providing default values that we may omit in the profiles
#------------------------------------------------------------------------------
c.BatchSpawnerBase.req_host = 'mesabi.xyz.edu'
c.BatchSpawnerBase.req_runtime = '12:00:00'
c.TorqueSpawner.state_exechost_exp = r'in-\1.mesabi.xyz.edu'
#------------------------------------------------------------------------------
# ProfilesSpawner configuration
#------------------------------------------------------------------------------
# List of profiles to offer for selection. Signature is:
#   List(Tuple( Unicode, Unicode, Type(Spawner), Dict ))
# corresponding to profile display name, unique key, Spawner class,
# dictionary of spawner config options.
#
# The first three values will be exposed in the input_template as {display},
# {key}, and {type}
#
c.ProfilesSpawner.profiles = [
   ( "Local server", 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
   ('Mesabi - 2 cores, 4 GB, 8 hours', 'mesabi2c4g12h', 'batchspawner.TorqueSpawner',
      dict(req_nprocs='2', req_queue='mesabi', req_runtime='8:00:00', req_memory='4gb')),
   ('Mesabi - 12 cores, 128 GB, 4 hours', 'mesabi128gb', 'batchspawner.TorqueSpawner',
      dict(req_nprocs='12', req_queue='ram256g', req_runtime='4:00:00', req_memory='125gb')),
   ('Mesabi - 2 cores, 4 GB, 24 hours', 'mesabi2c4gb24h', 'batchspawner.TorqueSpawner',
      dict(req_nprocs='2', req_queue='mesabi', req_runtime='24:00:00', req_memory='4gb')),
   ('Interactive Cluster - 2 cores, 4 GB, 8 hours', 'lab', 'batchspawner.TorqueSpawner',
      dict(req_nprocs='2', req_host='labhost.xyz.edu', req_queue='lab',
          req_runtime='8:00:00', req_memory='4gb', state_exechost_exp='')),
   ]
c.ProfilesSpawner.ip = '0.0.0.0'

Debugging batchspawner

Sometimes it can be hard to debug batchspawner, but it's not really once you know how the pieces interact. Check the following places for error messages:

Common problems:

Changelog

See CHANGELOG.md.