Home

Awesome

Code style: black Actions Status codecov.io Documentation Status Downloads


This is a PyPI mirror client according to PEP 381 + PEP 503 + PEP 691 http://www.python.org/dev/peps/pep-0381/.

bandersnatch maintainers are looking for more help! Please refer to our MAINTAINER documentation to see the roles and responsibilities. We would also ask you read our Mission Statement to ensure it aligns with your thoughts for this project.

Installation

The following instructions will place the bandersnatch executable in a virtualenv under bandersnatch/bin/bandersnatch.

Docker

This will pull latest build. Please use a specific tag if desired.

docker pull pypa/bandersnatch
docker run pypa/bandersnatch bandersnatch --help

Docker Compose

Bandersnatch setup using docker-compose is available here

pip

This installs the latest stable, released version.

python3 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help

Quickstart

Webserver

Configure your webserver to serve the web/ sub-directory of the mirror. For PEP691 support we need to respect the format the client requests.

For an nginx example, please look at our banderx docker container and nginx.conf example configuration.

For more information visit out official documentation for instructions on how to use a NGINX example Docker Image.

If you are looking to an docker-compose example head over here

Cron jobs

You need to set up one cron job to run the mirror itself.

Here's a sample that you could place in /etc/cron.d/bandersnatch:

    LC_ALL=en_US.utf8
    */2 * * * * root bandersnatch mirror |& logger -t bandersnatch[mirror]

This assumes that you have a logger utility installed that will convert the output of the commands to syslog entries.

SystemD Timers are also another alternative in today's modern world.

Maintenance

bandersnatch does not keep much local state in addition to the mirrored data. In general you can just keep rerunning bandersnatch mirror to make it fix errors.

If you want to force bandersnatch to check everything against the master PyPI:

Be aware that full syncs likely take hours depending on PyPI's performance and your network latency and bandwidth.

Other Commands

Operational notes

Case-sensitive filesystem needed

You need to run bandersnatch on a case-sensitive filesystem.

OS X natively does this OK even though the filesystem is not strictly case-sensitive and bandersnatch will work fine when running on OS X. However, tarring a bandersnatch data directory and moving it to, e.g. Linux with a case-sensitive filesystem will lead to inconsistencies. You can fix those by deleting the status files and have bandersnatch run a full check on your data.

Windows requires elevated prompt

Bandersnatch makes use of symbolic links. On Windows, this permission is turned off by default for non-admin users. In order to run bandersnatch on Windows either call it from an elevated command prompt (i.e. right-click, run-as Administrator) or give yourself symlink permissions in the group policy editor.

Many sub-directories needed

The PyPI has a quite extensive list of packages that we need to maintain in a flat directory. Filesystems with small limits on the number of sub-directories per directory can run into a problem like this:

    2013-07-09 16:11:33,331 ERROR: Error syncing package: zweb@802449
    OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'

Specifically we recommend to avoid using ext3. Ext4 and newer does not have the limitation of 32k sub-directories.

Client Compatibility

A bandersnatch static mirror is compatible only to the "static", cacheable parts of PyPI that are needed to support package installation. It does not support more dynamic APIs of PyPI that maybe be used by various clients for other purposes.

An example of an unsupported API is PyPI's XML-RPC interface, which is used when running pip search.

Bandersnatch Mission

The bandersnatch project strives to:

Contact

If you have questions or comments, please submit a bug report to https://github.com/pypa/bandersnatch/issues/new

Code of Conduct

Everyone interacting in the bandersnatch project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PSF Code of Conduct.

Kudos

This client is based on the original pep381client by Martin v. Loewis.

Richard Jones was very patient answering questions at PyCon 2013 and made the protocol more reliable by implementing some PyPI enhancements.

Christian Theune for creating and maintaining bandersnatch for many years!