Home

Awesome

python-blueprint

GitHub Actions Poetry Nox Ruff Type checked with mypy

Example Python project that demonstrates how to create a Python package using the latest Python testing, linting, and type checking tooling. The project contains a fact package that provides a simple implementation of the factorial algorithm (fact.lib) and a command line interface (fact.cli).

Package Management

This package uses Poetry to manage dependencies and isolated Python virtual environments.

To proceed, install Poetry globally onto your system.

Dependencies

Dependencies are defined in pyproject.toml and specific versions are locked into poetry.lock. This allows for exact reproducible environments across all machines that use the project, both during development and in production.

To install all dependencies into an isolated virtual environment:

[!TIP] Append --sync to uninstall dependencies that are no longer in use from the virtual environment.

$ poetry install

To activate the virtual environment that is automatically created by Poetry:

$ poetry shell

To deactivate the environment:

(fact) $ exit

To upgrade all dependencies to their latest versions:

$ poetry update

Packaging

This project is designed as a Python package, meaning that it can be bundled up and redistributed as a single compressed file.

Packaging is configured by:

To package the project as both a source distribution and a wheel:

$ poetry build

This will generate dist/fact-1.0.0.tar.gz and dist/fact-1.0.0-py3-none-any.whl.

[!TIP] Read more about the advantages of wheels to understand why generating wheel distributions are important.

Publish Distributions to PyPI

Source and wheel redistributable packages can be published to PyPI or installed directly from the filesystem using pip.

$ poetry publish

[!NOTE] To enable publishing, remove the "Private :: Do Not Upload" trove classifier.

Enforcing Code Quality

Automated code quality checks are performed using Nox and nox-poetry. Nox will automatically create virtual environments and run commands based on noxfile.py for unit testing, PEP 8 style guide checking, type checking and documentation generation.

[!NOTE] nox is installed into the virtual environment automatically by the poetry install command above. Run poetry shell to activate the virtual environment.

To run all default sessions:

(fact) $ nox

Unit Testing

Unit testing is performed with pytest. pytest has become the de facto Python unit testing framework. Some key advantages over the built-in unittest module are:

  1. Significantly less boilerplate needed for tests.
  2. PEP 8 compliant names (e.g. pytest.raises() instead of self.assertRaises()).
  3. Vibrant ecosystem of plugins.

pytest will automatically discover and run tests by recursively searching for folders and .py files prefixed with test for any functions prefixed by test.

The tests folder is created as a Python package (i.e. there is an __init__.py file within it) because this helps pytest uniquely namespace the test files. Without this, two test files cannot be named the same, even if they are in different subdirectories.

Code coverage is provided by the pytest-cov plugin.

When running a unit test Nox session (e.g. nox -s test), an HTML report is generated in the htmlcov folder showing each source file and which lines were executed during unit testing. Open htmlcov/index.html in a web browser to view the report. Code coverage reports help identify areas of the project that are currently not tested.

pytest and code coverage are configured in pyproject.toml.

To pass arguments to pytest through nox:

(fact) $ nox -s test -- -k invalid_factorial

Code Style Checking

PEP 8 is the universally accepted style guide for Python code. PEP 8 code compliance is verified using Ruff. Ruff is configured in the [tool.ruff] section of pyproject.toml.

Some code style settings are included in .editorconfig and will be configured automatically in editors such as PyCharm.

To lint code, run:

(fact) $ nox -s lint

To automatically fix fixable lint errors, run:

(fact) $ nox -s lint_fix

Automated Code Formatting

Ruff is used to automatically format code and group and sort imports.

To automatically format code, run:

(fact) $ nox -s fmt

Type Checking

Type annotations allows developers to include optional static typing information to Python source code. This allows static analyzers such as mypy, PyCharm, or Pyright to check that functions are used with the correct types before runtime.

Editors such as PyCharm and VS Code are able to provide much richer auto-completion, refactoring, and type checking while the user types, resulting in increased productivity and correctness.

def factorial(n: int) -> int:
    ...

mypy is configured in pyproject.toml. To type check code, run:

(fact) $ nox -s type_check

See also awesome-python-typing.

Distributing Type Annotations

PEP 561 defines how a Python package should communicate the presence of inline type annotations to static type checkers. mypy's documentation provides further examples on how to do this.

Mypy looks for the existence of a file named py.typed in the root of the installed package to indicate that inline type annotations should be checked.

Continuous Integration

Continuous integration is provided by GitHub Actions. This runs all tests, lints, and type checking for every commit and pull request to the repository.

GitHub Actions is configured in .github/workflows/python.yml.

Documentation

Generating a User Guide

Material for MkDocs is a powerful static site generator that combines easy-to-write Markdown, with a number of Markdown extensions that increase the power of Markdown. This makes it a great fit for user guides and other technical documentation.

The example MkDocs project included in this project is configured to allow the built documentation to be hosted at any URL or viewed offline from the file system.

To build the user guide, run,

(fact) $ nox -s docs

and open docs/user_guide/site/index.html using a web browser.

To build the user guide, additionally validating external URLs, run:

(fact) $ nox -s docs_check_urls

To build the user guide in a format suitable for viewing directly from the file system, run:

(fact) $ nox -s docs_offline

To build and serve the user guide with automatic rebuilding as you change the contents, run:

(fact) $ nox -s docs_serve

and open http://127.0.0.1:8000 in a browser.

Each time the main Git branch is updated, the .github/workflows/pages.yml GitHub Action will automatically build the user guide and publish it to GitHub Pages. This is configured in the docs_github_pages Nox session. This hosted user guide can be viewed at https://johnthagen.github.io/python-blueprint/.

Generating API Documentation

This project uses mkdocstrings plugin for MkDocs, which renders Google-style docstrings into an MkDocs project. Google-style docstrings provide a good mix of easy-to-read docstrings in code as well as nicely-rendered output.

"""Computes the factorial through a recursive algorithm.

Args:
    n: A positive input value.

Raises:
    InvalidFactorialError: If n is less than 0.

Returns:
    Computed factorial.
"""

Project Structure

Traditionally, Python projects place the source for their packages in the root of the project structure, like:

fact
├── fact
│   ├── __init__.py
│   ├── cli.py
│   └── lib.py
├── tests
│   ├── __init__.py
│   └── test_fact.py
├── noxfile.py
└── pyproject.toml

However, this structure is known to have bad interactions with pytest and nox, two standard tools maintaining Python projects. The fundamental issue is that Nox creates an isolated virtual environment for testing. By installing the distribution into the virtual environment, nox ensures that the tests pass even after the distribution has been packaged and installed, thereby catching any errors in packaging and installation scripts, which are common. Having the Python packages in the project root subverts this isolation for two reasons:

  1. Calling python in the project root (for example, python -m pytest tests/) causes Python to add the current working directory (the project root) to sys.path, which Python uses to find modules. Because the source package fact is in the project root, it shadows the fact package installed in the Nox session.
  2. Calling pytest directly anywhere that it can find the tests will also add the project root to sys.path if the tests folder is a Python package (that is, it contains a __init__.py file). pytest adds all folders containing packages to sys.path because it imports the tests like regular Python modules.

In order to properly test the project, the source packages must not be on the Python path. To prevent this, there are three possible solutions:

  1. Remove the __init__.py file from tests and run pytest directly as a Nox session.
  2. Remove the __init__.py file from tests and change the working directory of python -m pytest to tests.
  3. Move the source packages to a dedicated src folder.

The dedicated src directory is the recommended solution by pytest when using Nox and the solution this blueprint promotes because it is the least brittle even though it deviates from the traditional Python project structure. It results is a directory structure like:

fact
├── src
│   └── fact
│       ├── __init__.py
│       ├── cli.py
│       └── lib.py
├── tests
│   ├── __init__.py
│   └── test_fact.py
├── noxfile.py
└── pyproject.toml

Licensing

Licensing for the project is defined in:

This project uses a common permissive license, the MIT license.

You may also want to list the licenses of all the packages that your Python project depends on. To automatically list the licenses for all dependencies in (and their transitive dependencies) using pip-licenses:

(fact) $ nox -s licenses
...
 Name      Version  License     
 click     8.1.3    BSD License 
 colorama  0.4.4    BSD License 
 typer     0.4.1    MIT License

Container

Docker is a tool that allows for software to be packaged into isolated containers. It is not necessary to use Docker in a Python project, but for the purposes of presenting best practice examples, a Docker configuration is provided in this project. The Docker configuration in this repository is optimized for small size and increased security, rather than simplicity.

Docker is configured in:

To build the container image:

$ docker build --tag fact .

To run the image in a container:

$ docker run --rm --interactive --tty fact 5

[!NOTE] If you need to install Poetry on Alpine Linux, see the pre-built poetry package for that platform rather than running pip install poetry. This avoids needing to build Poetry dependencies from source.

FROM alpine:latest

RUN apk add --no-cache poetry

Miscellaneous

Shebang Line

The proper shebang line for Python scripts is:

#!/usr/bin/env python3

Installing Newer Python on Ubuntu

Ubuntu releases come with a default python3 executable. This is frozen for the life of the OS and only receives security and bug fixes. To install a newer version of Python globally, consider the deadsnakes PPA.

sudo add-apt-repository ppa:deadsnakes
sudo apt update
sudo apt install python3.12

Package Dependency Tree

poetry show is a Poetry command for listing installed packages. The --tree option displays them in the form of a dependency tree. For large projects, it is often difficult to determine dependency relationships soley from manually inspecting poetry.lock.

$ poetry show --tree --only main

rich 12.6.0 Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
├── commonmark >=0.9.0,<0.10.0
├── pygments >=2.6.0,<3.0.0
└── typing-extensions >=4.0.0,<5.0
typer 0.7.0 Typer, build great CLIs. Easy to code. Based on Python type hints.
├── click >=7.1.1,<9.0.0
│   └── colorama * 
├── colorama >=0.4.3,<0.5.0
├── rich >=10.11.0,<13.0.0
│   ├── commonmark >=0.9.0,<0.10.0 
│   ├── pygments >=2.6.0,<3.0.0 
│   └── typing-extensions >=4.0.0,<5.0 
└── shellingham >=1.3.0,<2.0.0

PyCharm Configuration

[!TIP] Looking for a vivid dark color scheme for PyCharm? Try One Dark theme.

To configure PyCharm to align to the code style used in this project:

Ruff Integration

Integrate Ruff linting and formatting into PyCharm.

Linting and Formatting

  1. Install the Ruff PyCharm Plugin
  2. Open Preferences or Settings | Tools | Ruff
    • Check: Run Ruff when the python file is saved
    • Check: Use ruff format

Now, on <kbd>ctrl+s</kbd>, the current source file will be automatically formatted on save.

Import Sorting

  1. Ensure that the File Watchers Plugin is installed.
  2. Open Preferences or Settings | Tools | File Watchers and select + | <custom>

  1. Fill in the following fields
    • Name: ruff check --select I --fix
    • File Type: Python
    • Scope: Project Files
    • Program: $PyInterpreterDirectory$/python
    • Arguments: -m ruff check --select I --fix $FilePath$
    • Output paths to refresh: $FilePath$
    • Working directory: $ProjectFileDir$
    • Advanced Options
      • Uncheck: Auto-save edited files to trigger the watcher
      • Uncheck: Trigger the watcher on external changes

[!TIP] These tools work best if you properly mark directories as excluded from the project that should be, such as .nox. See https://www.jetbrains.com/help/pycharm/project-tool-window.html#content_pane_context_menu on how to Right-Click | Mark Directory as | Excluded.

Nox Support

PyCharm does not yet natively support Nox. The recommended way to launch Nox from PyCharm is to create a Python Run Configuration.