Awesome
Protocol Buffers for the Rest of Us
Motivation
Are you annoyed by having to fix the absolute imports generated by protoc
?
If so, then protoletariat
is the tool for you.
protoletariat
has one goal: fixing the broken imports for the Python code
generated by protoc
.
See https://github.com/protocolbuffers/protobuf/issues/1491 for the discussion that inspired this tool.
Installation
Artifact | Status | Installation Command |
---|---|---|
PyPI Package | pip install protoletariat | |
Conda Package | conda install protoletariat -c conda-forge | |
Docker Image | ∅ | docker pull ghcr.io/cpcloud/protoletariat:latest |
Note: the conda-forge
package version may lag behind the other artifacts by a few hours.
Usage
protoletariat
is designed to be run as a post-processing step after running
protoc
. It operates directly on the generated code.
Example
Here's an example of how to use the tool, called protol
:
- Create a few protobuf files
// thing1.proto
syntax = "proto3";
import "thing2.proto";
package things;
message Thing1 {
Thing2 thing2 = 1;
}
// thing2.proto
syntax = "proto3";
package things;
message Thing2 {
string data = 1;
}
- Run
protoc
on those files
$ mkdir out
$ protoc \
--python_out=out \
--proto_path=directory/containing/protos thing1.proto thing2.proto
- Run
protol
on the generated code
$ protol \
--create-package \
--in-place \
--python-out out \
protoc --proto-path=directory/containing/protos thing1.proto thing2.proto
The out/thing1_pb2.py
module should show a diff containing at least these lines:
-import thing2_pb2 as thing2__pb2
-
+from . import thing2_pb2 as thing2__pb2
How it works
At a high level protoletariat
converts absolute imports to relative imports.
However, it doesn't convert just any absolute import to a relative import.
The protol
tool will only convert imports that were generated from .proto
files. It does this by inspecting FileDescriptorProtos
generated from the
proto files.
The core rewrite mechanism is implemented using a simplified form of pattern matching that looks at the Python AST, and invokes rewrite rules for matched import patterns.
Subcommands
protoletariat
has a subcommand for each tool that you might like to use to
generate FileDescriptorSet
bytes:
Subcommand | Description |
---|---|
protoc | Uses protoc to generate FileDescriptorSet bytes |
buf | Uses buf to generate FileDescriptorSet bytes |
raw | You provide the FileDescriptorSet bytes as a file or directly from stdin |
Help
$ protol
Usage: protol [OPTIONS] COMMAND [ARGS]...
Rewrite protoc or buf-generated imports for use by the protoletariat.
Options:
-o, --python-out DIRECTORY Directory containing protoc or buf-generated Python code [required]
--in-place / --not-in-place Overwrite all relevant files under `--python-out` with adjusted imports [default: not-in-place]
--create-package / --dont-create-package
Recursively create __init__.py files under `--python-out` [default: dont-create-package]
-s, --module-suffixes TEXT Suffixes of Python/mypy modules to process [default: _pb2.py, _pb2.pyi, _pb2_grpc.py, _pb2_grpc.pyi]
--exclude-google-imports / --dont-exclude-google-imports
Exclude rewriting imports prefixed with google/protobuf
-e, --exclude-imports-glob TEXT
Exclude imports matching a glob pattern from being rewritten. Multiple values are allowed
--help Show this message and exit.
Commands:
buf Use buf to generate the FileDescriptorSet blob
protoc Use protoc to generate the FileDescriptorSet blob
raw Rewrite imports using FileDescriptorSet bytes from a file or stdin