Awesome
JSON:API/OpenAPI Test Server
Requirements
Server
The test server requires a Postgres database and Redis instace.
Usage
The test server and commandline tool will execute tests against an OpenAPI document. Out of box, you get errors for unprocessable OpenAPI documentation, warnings for certain unhandled types of APIs, and a test that every OpenAPI Example parses under the corresponding request/response body schema.
By default the server will assume that all request and response bodies are JSON:API compliant. The server will warn you and fall back to a non-JSON:API (but still JSON) request/response parsing mode if it fails to generate a test based on a JSON:API compliant schema. You can also opt out of attempting to interpret any particular OpenAPI Media Item as JSON:API with the x-not-json-api: true
specification extension. This gets added to the OpenAPI document inside the Media Item but outside the schema.
For example,
...
{
"content": {
"application/json": {
"x-not-json-api": true,
"schema": {
"type": "object"
}
}
}
}
...
You can additionally create tests that make API calls and verify that the actual responses from your server are parseable under the corresponding response schema. You do this with the x-tests
Specification Extension on the OpenAPI Media Type Object within a Response Object (e.g. responses/'200'/content/'application/json'/x-tests
). x-tests
has the following structure:
{
"test_name": {
"test_host": "url",
"skip_example": false,
"ignore_missing_parameter_warnings": false,
"parameters": {
"path_param_name": "value",
"header_param_name": "value"
},
"query_parameters": [
{
"name": "param_name",
"value": "param_value"
}
]
}
}
Parameters:
test_host
: optional, if omitted then default server for API will be used.skip_example
: optional, defaults to false.ignore_missing_parameter_warnings
: optional, defaults to false.parameters
values: Must be strings, even if the parameter type is Int or other.
The Commandline Tool
The command line tool's usage can be printed with --help
and it is as follows:
OVERVIEW: Build and run tests based on an OpenAPI Document.
USAGE: APITest [--dump-files <directory path>] [--fail-hard] [--ignore-warnings] [--validate-all] [--openapi-file <file path>] [--override-server <url>]
OPTIONS:
--dump-files <directory path>
Dump produced test files in a zipped file at the specified location.
Tip: A good location to dump files is "./out". For the Dockerized tool this will be `/app/out` and when running the tool natively on your machine this will be the `out` folder relative to the current working directory.
Not using this argument will result in test files being deleted after execution of the tests.
-f, --fail-hard Produce a non-zero exit code if any tests fail.
--ignore-warnings Do not print warnings in the output.
--validate-all Perform validation and linting on the OpenAPI documentation in addition to generating tests from it.
--openapi-file <file path>
Specify a filename from the local filesystem from which to read OpenAPI documentation.
Alternatively, set the `API_TEST_IN_FILE` environment variable.
Either the environment variable or this argument must be used to indicate the OpenAPI file from which the tests should be generated.
--override-server <url> Override the server definition(s) in the OpenAPI document for the purposes of this test run.
This argument allows you to make API requests against a different server than the input OpenAPI documentation specifies for this test run.
Not using this argument will result in the API server options from the OpenAPI documentation being used.
-h, --help Show help information.
Against a URL
You can point the test tool at a URL serving up OpenAPI documentation. The URL can either require HTTP Basic Authentication or no authentication.
The unauthenticated version only requires the API_TEST_IN_URL
environment variable.
docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_URL=https://website.com/api/documentation' mattpolzin2/api-test-server
The authenticated version additionally requires the API_TEST_USERNAME
and API_TEST_PASSWORD
environment variables.
docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_USERNAME=username' --env 'API_TEST_PASSWORD=password' mattpolzin2/api-test-server ./APITest
Against a local file
You can point the test tool at a local file if you mount that file into the docker container and specify the mount destination with the API_TEST_IN_FILE
environment variable or the --openapi-file
option for the test
command.
# command option
docker run --rm --entrypoint ./APITest -v '/full/path/to/openapi.json:/api/openapi.json' mattpolzin2/api-test-server --openapi-file /api/openapi.json
# ENV var
docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_FILE=/api/openapi.json' -v '/full/path/to/openapi.json:/api/openapi.json' mattpolzin2/api-test-server
Note that you cannot use relative paths with bind mounts but if, for example, your openapi.json
file is in the current working directory then you could invoke as:
docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_FILE=/api/openapi.json' -v "$(pwd)/openapi.json:/api/openapi.json" mattpolzin2/api-test-server
API Host Override
You can specify an override test server URL if you want to make API test requests against a different URL than is specified by the OpenAPI documentation. You use the test
command's --override-server
option for this.
docker run --rm --entrypoint ./APITest -v '/full/path/to/openapi.json:/api/openapi.json' mattpolzin2/api-test-server --openapi-file /api/openapi.json --override-server https://test.server.com
Dumping test files
The test tool works by generating Swift code to parse examples and test responses. These test files include JSON:API models that could be used as a basis for client implementations. You can dump the test files with the --dump-files
argument to the ./APITest test
command. You must also mount the output directory (or don't remove the container and then docker cp
later) so you can access the generated file from outside of the container.
docker run --rm --env 'API_TEST_IN_URL=https://website.com/api/documentation' -v "$(pwd)/out:/app/out" mattpolzin2/api-test-server ./APITest --dump-files /app/out
You will find the dumped files at /app/out/api_test_files.zip
. TIP: You can also find the raw text logs from a test run at /app/out/api_test.log
.
The Test Server
You can run an API Test server that accepts requests to run tests at HTTP endpoints. This requires the same input file or URL environment variables explained in the above section but you also must provide a Postgres database for the server to use as its persistence layer. You specify this database using a Postgres URL in the API_TEST_DATABASE_URL
environment variable. A Redis instance is required to queue up the test runs. You specify the Redis URL in the API_TEST_REDIS_URL
environment variable.
First you need to run the migrator against your Postgres database.
docker run --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_DATABASE_URL=postgres://user:password@host:port/databasename' --env 'API_TEST_REDIS_URL=redis://host:port' -p '8080:80' mattpolzin2/api-test-server migrate --yes
Then you can start the server.
docker run --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_DATABASE_URL=postgres://user:password@host:port/databasename' --env 'API_TEST_REDIS_URL=redis://host:port' -p '8080:80' mattpolzin2/api-test-server
Jobs Queue
Testing is run in a jobs queue. That queue can be run in the same process as the API server if you specify API_TEST_IN_PROCESS_QUEUES=true
as an environment variable but the recommendation is to run the jobs service as its own process.
You start the Jobs Queue using the same docker image as the server but you specify the queues
command.
docker run --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_DATABASE_URL=postgres://user:password@host:port/databasename' --env 'API_TEST_REDIS_URL=redis://host:port' mattpolzin2/api-test-server queues
NOTE You must explicitly expose the port to the host device. In this example, http://localhost:8080
will point to the server which is listening on port 80
in the container.
Visit the /docs
API endpoint to see what endpoints the server provides.
Building
Note that Vapor 4 (and therefore this server) requires Swift 5.2.
Running and Testing
As of this writing, you need to run swift package generate-xcodeproj
and then open that project in Xcode. Using Xcode's built-in Swift Package Manager support is currently broken for libraries like swift-syntax that require dynamic libraries from the Swift toolchain. swift build
, swift test
, etc. from the command line will work fine, though.
Generating API Documentation
To generate API documentation, run the GenAPIDocumenation
target and save the output to Public/openapi.yml
. This file is not committed to the repository but it will be picked up by the ReDoc UI served at the /docs
API endpoint.
Documentation is generated as part of the Docker image build so you do not need to perform it as a separate step if you are building the Docker image anyway.
Building Docker Image
From the root folder of the repository, run
docker build -t api-test-server:latest .
Once done, you will have the api-test-server:latest
Docker image.