Awesome

VS Code Test Adapter API

This package contains the APIs that Test Adapters or Test Controllers need to implement to work with the VS Code Test Explorer.

The API reference documentation can be found here.

Implementing a Test Adapter

A Test Adapter allows the Test Explorer to load and run the tests using a particular test framework. There is an Example Test Adapter that you can use as a template for your implementation. It uses the Test Adapter Util package for some standard tasks (mostly logging and registering the adapter with Test Explorer).

Registering your Test Adapter with the Test Explorer

The easiest way to register your Test Adapter with the Test Explorer is to use the TestAdapterRegistrar class from the Test Adapter Util package in the activate() function of your extension:

import { TestHub, testExplorerExtensionId } from 'vscode-test-adapter-api';
import { TestAdapterRegistrar } from 'vscode-test-adapter-util';

export function activate(context: vscode.ExtensionContext) {

    const testExplorerExtension = vscode.extensions.getExtension<TestHub>(testExplorerExtensionId);

    if (testExplorerExtension) {

        const testHub = testExplorerExtension.exports;

        context.subscriptions.push(new TestAdapterRegistrar(
            testHub,
            workspaceFolder => new MyTestAdapter(workspaceFolder)
        ));
    }
}

The TestAdapterRegistrar will create and register one Test Adapter per workspace folder and unregister it when the workspace folder is closed or removed from the workspace. It requires the Test Adapter to implement a dispose() method that will be called after unregistering the Test Adapter. The TestAdapterRegistrar has a dispose() method itself which should be called when the Test Adapter extension is deactivated. In the example above, we achieve this by adding it to context.subscriptions.

While most Test Adapter implementations create one adapter per workspace folder, this is not a requirement. If you don't want to do this or have any other reason to not use TestAdapterRegistrar, you can easily register your adapter like this:

const testAdapter = new MyTestAdapter(...);
testHub.registerTestAdapter(testAdapter);

Don't forget to unregister your adapter when your extension is deactivated:

testHub.unregisterTestAdapter(testAdapter);

Implementing the events

Every Test Adapter needs to implement the tests and testStates events and it's recommended to also implement the retire event:

private readonly testsEmitter = new vscode.EventEmitter<TestLoadStartedEvent | TestLoadFinishedEvent>();
private readonly testStatesEmitter = new vscode.EventEmitter<TestRunStartedEvent | TestRunFinishedEvent | TestSuiteEvent | TestEvent>();
private readonly retireEmitter = new vscode.EventEmitter<RetireEvent>();

get tests(): vscode.Event<TestLoadStartedEvent | TestLoadFinishedEvent> {
    return this.testsEmitter.event;
}
get testStates(): vscode.Event<TestRunStartedEvent | TestRunFinishedEvent | TestSuiteEvent | TestEvent> {
    return this.testStatesEmitter.event;
}
get retire(): vscode.Event<RetireEvent> {
    return this.retireEmitter.event;
}

Loading the tests

The load() method is responsible for loading the tests and sending their metadata to the Test Explorer. This method will be called by the Test Explorer, but you should also consider running it automatically whenever one of the test files has changed (as described below).

It uses the tests event to communicate with the Test Explorer. It must send one TestLoadStartedEvent at the beginning and one TestLoadFinishedEvent at the end.

• If the adapter found some tests, these must be sent using TestLoadFinishedEvent#suite
• If the adapter ran successfully but didn't find any tests, the TestLoadFinishedEvent should contain no suite or errorMessage
• If the adapter failed to load the tests (e.g. due to some misconfiguration by the user) and you want to show a message to the user that might help him fix the problem, that message must be sent using TestLoadFinishedEvent#errorMessage
• It is important that you ensure that both events are sent (each one exactly once) when loading the tests, otherwise you will confuse the Test Explorer
• Furthermore, you should make sure that the load() method doesn't run twice in parallel
• The TestInfo#id needs to be unique for each test because otherwise the Test Explorer won't know which test to assign the test results to
• The TestSuiteInfo#label of the root suite sent to the Test Explorer is not shown when there is only one adapter, but when there are multiple adapters, it will be shown to let the user know where the tests are coming from, so it is recommended to use the name of the testing framework that your Test Adapter supports as the root suite label
• After sending the TestLoadFinishedEvent you should also send a RetireEvent to mark the test states as outdated. If your Test Adapter doesn't contain the retire property, the Test Explorer will automatically retire all test states after a TestLoadFinishedEvent.

Here's a skeleton for a typical implementation of load():

private isLoading = false;

async load(): Promise<void> {

    if (this.isLoading) return; // it is safe to ignore a call to `load()`, even if it comes directly from the Test Explorer

    this.isLoading = true;
    this.testsEmitter.fire({ type: 'started' });

    try {

        const suite = ... // load the tests, the result may be `undefined`...

        this.testsEmitter.fire({ type: 'finished', suite });

    } catch (e) {
        this.testsEmitter.fire({ type: 'finished', errorMessage: util.inspect(e));
    }

    this.retireEmitter.fire({}); 

    this.isLoading = false;
}

Running the tests

The run() method runs the tests in a child process and sends the results to the Test Explorer using the testStates event. It must first send a TestRunStartedEvent containing the IDs of the tests that it is going to run. Next, it should send events for all tests and suites being started or completed. Finally, it must send a TestRunFinishedEvent. Technically you could run this method automatically (just like you can do with the load() method), but this is usually not recommended, if the user wants tests to be run automatically, he should use the autorun feature of the Test Explorer.

For example, if there is one test suite with ID suite1 containing one test with ID test1, a successful test run would usually emit the following events:

{ type: 'started', tests: ['suite1'] }
{ type: 'suite', suite: 'suite1', state: 'running' }
{ type: 'test', test: 'test1', state: 'running' }
{ type: 'test', test: 'test1', state: 'passed' }
{ type: 'suite', suite: 'suite1', state: 'completed' }
{ type: 'finished' }

• The TestSuiteEvents are optional, currently they are only needed if you want to add tests or suites during a test run
• The TestEvents with state: 'running' are also optional but recommended so that the user can see which test is currently running
• The TestEvents can be sent in any order, in particular having multiple tests' states set to 'running' at the same time is supported (useful if the test framework supports running multiple tests in parallel)
• It is important that you ensure that the TestRunStartedEvent is the first event and the TestRunFinishedEvent is the last event sent when running the tests (and each is sent exactly once), otherwise you will confuse the Test Explorer
• Furthermore, you should make sure that the run() method doesn't run twice in parallel

Here's a skeleton for a typical implementation of run() and cancel():

private runningTestProcess: : child_process.ChildProcess | undefined;

run(testsToRun: string[]): Promise<void> {

    if (this.runningTestProcess !== undefined) return; // it is safe to ignore a call to `run()`

    this.testStatesEmitter.fire({ type: 'started', tests: testsToRun });

    return new Promise<void>((resolve, reject) => {

        this.runningTestProcess = child_process.spawn(...);

        // we will _always_ receive an `exit` event when the child process ends, even if it crashes or
        // is killed, so this is a good place to send the `TestRunFinishedEvent` and resolve the Promise
        this.runningTestProcess.once('exit', () => {

            this.runningTestProcess = undefined;
            this.testStatesEmitter.fire({ type: 'finished' });
            resolve();

        });
    });
}

cancel(): void {
    if (this.runningTestProcess !== undefined) {
        this.runningTestProcess.kill();
        // there is no need to do anything else here because we will receive an `exit` event from the child process
    }
}

• In this example we use child_process.spawn() to start the child process, but if the child process runs javascript, you may want to use child_process.fork()
• Using child_process.exec() or child_process.execFile() is not recommended because these functions buffer stdout and stderr, which has some downsides:
  • you won't be able to access the output of the child process until it has finished (i.e. you have to wait until the entire test run is completed before you can show the results to the user)
  • the buffers are limited in size and the child process (and hence the test run) will be terminated when this limit is reached
• Don't use any of the synchronous methods: these would block the Extension Host process (and hence all VS Code extensions) until the test run is finished

Watching for changes in the workspace

When a VS Code setting or a source file in the workspace that influences the tests changes, the user can manually reload and/or rerun the tests, but ideally the Test Adapter should do this automatically for him.

To watch for changes in the VS Code settings, use the onDidChangeConfiguration() event.

For file changes, there are several options:

• The onDidSaveTextDocument() event is fired by VS Code when the user saves his changes to a text document. Obviously, this is only useful if you're not interested in files that are generated/changed by another process (like a compiler/transpiler)
• You can use createFileSystemWatcher() to watch for file changes in the workspace. The only downside is that this will not watch files that the user has hidden using the files.exclude setting (so this may also miss changes to compiled/transpiled files if the user chose to hide them in VS Code)
• You can use chokidar to watch for file changes. This is similar to using createFileSystemWatcher() (and VS Code also uses chokidar to implement its FileSystemWatcher), but it also lets you watch files that are hidden in VS Code. The main downside is that this is relatively resource-intensive, so you should make sure that you only watch the files that you definitely need to watch (when you use createFileSystemWatcher(), you're "freeloading" on the watcher that VS Code has already set up anyway, so that doesn't use any additional resources)
• If you're watching for changes in files generated by a compiler/transpiler, you may get multiple events in quick succession and should consider using some debouncing mechanism

To reload the tests after some change, you can simply call your load() method yourself. If the tests themselves haven't changed but their states may be outdated (e.g. if the change was in a source file for the application being tested), you can send a RetireEvent to the Test Explorer. This event will also trigger a test run if the user has enabled the autorun feature for the tests. If you know that the change may only affect the states of some of the tests, you can send the IDs of these tests in the RetireEvent, only those tests will be retired (marked as outdated). Otherwise you can simply send an empty object as the RetireEvent (as in the example below), this will retire all tests.

This example watches for changes to a VS Code setting myTestAdapter.testFiles and reloads the tests when the setting is changed:

vscode.workspace.onDidChangeConfiguration(configChange => {
    if (configChange.affectsConfiguration('myTestAdapter.testFiles', this.workspaceFolder.uri)) {
        this.load();
    }
});

This example uses onDidSaveTextDocument() to watch for file changes:

vscode.workspace.onDidSaveTextDocument(document => {
    if (isTestFile(document.uri)) {
        // the changed file contains tests, so we reload them
        this.load();
    } else if (isApplicationFile(document.uri)) {
        // the changed file is part of the application being tested, so the test states may be out of date
        this.retireEmitter.fire({});
    }
});