Home

Awesome

Interop Test Runner

The Interop Test Runner aims to automatically generate an interop matrix by running multiple test cases using different QUIC implementations.

Requirements

The Interop Runner is written in Python 3. You'll need to install the following softwares to run the interop test:

pip3 install -r requirements.txt

Running the Interop Runner

Run the interop tests:

python3 run.py

IPv6 support

To enable IPv6 support for the simulator on Linux, the ip6table_filter kernel module needs to be loaded on the host. If it isn't loaded on your machine, you'll need to run sudo modprobe ip6table_filter.

Building a QUIC endpoint

To include your QUIC implementation in the Interop Runner, create a Docker image following the instructions for setting up an endpoint in the quic-network-simulator, publish it on Docker Hub and add it to implementations.json. Once your implementation is ready to interop, please send us a PR with this addition. Read on for more instructions on what to do within the Docker image.

Typically, a test case will require a server to serve files from a directory, and a client to download files. Different test cases will specify the behavior to be tested. For example, the Retry test case expects the server to use a Retry before accepting the connection from the client. All configuration information from the test framework to your implementation is fed into the Docker image using environment variables. The test case is passed into your Docker container using the TESTCASE environment variable. If your implementation doesn't support a test case, it MUST exit with status code 127. This will allow us to add new test cases in the future, and correctly report test failures und successes, even if some implementations have not yet implented support for this new test case.

The Interop Runner mounts the directory /www into your server Docker container. This directory will contain one or more randomly generated files. Your server implementation is expected to run on port 443 and serve files from this directory. Equivalently, the Interop Runner mounts /downloads into your client Docker container. The directory is initially empty, and your client implementation is expected to store downloaded files into this directory. The URLs of the files to download are passed to the client using the environment variable REQUESTS, which contains one or more URLs, separated by a space.

After the transfer is completed, the client container is expected to exit with exit status 0. If an error occurred during the transfer, the client is expected to exit with exit status 1. After completion of the test case, the Interop Runner will verify that the client downloaded the files it was expected to transfer, and that the file contents match. Additionally, for certain test cases, the Interop Runner will use the pcap of the transfer to verify that the implementations fulfilled the requirements of the test (for example, for the Retry test case, the pcap should show that a Retry packet was sent, and that the client used the Token provided in that packet).

The Interop Runner generates a key and a certificate chain and mounts it into /certs. The server needs to load its private key from priv.key, and the certificate chain from cert.pem.

Examples

If you're not familiar with Docker, it might be helpful to have a look at the Dockerfiles and scripts that other implementations use:

Implementers: Please feel free to add links to your implementation here!

Note that the online interop runner requires linux/amd64 architecture, so if you build on a different architecture (e.g. "Apple silicon"), you would need to use --platform linux/amd64 with docker build to create a compatible image. Even better, and the recommended approach, is to use a multi-platform build to provide both amd64 and arm64 images, so everybody can run the interop locally with your implementation. To build the multi-platform image, you can use the docker buildx command:

docker buildx create --use
docker buildx build --pull --push --platform linux/amd64,linux/arm64 -t <name:tag> .

Logs

To facilitate debugging, the Interop Runner saves the log files to the logs directory. This directory is overwritten every time the Interop Runner is executed.

The log files are saved to a directory named #server_#client/#testcase. output.txt contains the console output of the interop test runner (which might contain information why a test case failed). The server and client logs are saved in the server and client directory, respectively. The sim directory contains pcaps recorded by the simulator.

If implementations wish to export the TLS secrets, they are encouraged to do so in the format in the NSS Key Log format. The interop runner sets the SSLKEYLOGFILE environment variable to a file in the logs directory. In the future, the interop runner might use those files to decode the traces.

Implementations that implement qlog should export the log files to the directory specified by the QLOGDIR environment variable.

Test cases

The Interop Runner implements the following test cases. Unless noted otherwise, test cases use HTTP/0.9 for file transfers. More test cases will be added in the future, to test more protocol features. The name in parentheses is the value of the TESTCASE environment variable passed into your Docker container.