Home

Awesome

@jobber/jest-a-coverage-slip-detector

License: MIT npm version

This library ensures that new files have Jest coverage meeting the configured goals.

Additionally, this library can be added to an existing project such that legacy files not meeting the coverage goals are added to an exception list where they raise an error if coverage slips, and ratchet upwards as progress is made improving them, all while enforcing the higher coverage goals on net new code.

Requirements

Installation

npm install --save-dev @jobber/jest-a-coverage-slip-detector

Configure Jest

Within jest.config.js or jest.config.ts:

  1. Ensure Jest is configured to include json in coverageReporters.
  2. Ensure that coverage collection is enabled in the CI command (e.g. with the --coverage parameter).
  3. Either remove the coverageThreshold configuration from Jest, or set it to: coverageThreshold: { global: {} }.
  4. Wrap the configuration with the withJestSlipDetection utility method in order to dynamically leverage collectCoverageFrom set to the configured coverageGlob.

Example (JavaScript):

const { withJestSlipDetection } = require("@jobber/jest-a-coverage-slip-detector");

module.exports = withJestSlipDetection({
  coverageReporters: [
    "json" // plus any other reporters, e.g. "lcov", "text", "text-summary"
  ],
  coverageThreshold: { global: {} },
});

Example (TypeScript):

import type { Config } from "@jest/types";
import { withJestSlipDetection } from "@jobber/jest-a-coverage-slip-detector";

const config: Config.InitialOptions = {
  coverageReporters: [
    "json" // plus any other reporters, e.g. "lcov", "text", "text-summary"
  ],
  transform: {
    "^.+\\.ts?$": "ts-jest",
  }
};

export default withJestSlipDetection(config);

Configure Scripts

These scripts assume you have the following two reporters installed:

npm i -D jest-progress-bar-reporter jest-junit

Within package.json:

{
  "scripts": {
    "test": "jest",
    "test:ci": "jest --runInBand --coverage --reporters=jest-progress-bar-reporter --reporters=jest-junit --ci",
    "posttest:ci": "npm run test:validateCoverage",
    "test:generateCoverage": "jest --coverage --reporters=jest-progress-bar-reporter --ci",
    "test:validateCoverage": "jest-a-coverage-slip-detector",
    "test:updateCoverageExceptions": "jest-a-coverage-slip-detector --update", // Used to 'ratchet' up coverage after improving it.
    "test:setCoverageExceptionsBaseline": "jest-a-coverage-slip-detector --force-update" // Sets the baseline for test coverage (accepts any under-target coverage).
  }
}

Configure Coverage Goals

If you're happy with the defaults below, nothing further is needed:

{
  "coverageGoal": { "lines": 80, "functions": 80, "statements": 80, "branches": 80 },
  "coverageGlob": [
    "**/*.{ts,tsx,js,jsx}",
    "!**/node_modules/**",
    "!**/vendor/**",
  ]
}

Otherwise:

Example:

{
  "coverageGoal": { "lines": 90, "functions": 90, "statements": 90, "branches": 90 },
  "coverageGlob": ["./app/javascript/**/*.{ts,tsx,js,jsx}"]
}

Usage

First Run

  1. Generate and view coverage errors: npm run test:generateCoverage && npm run test:validateCoverage
  2. Snapshot current coverage errors as legacy exceptions: npm run test:setCoverageExceptionsBaseline
  3. Commit the generated exception listing (generatedCoverageExceptions.json by default) to source control
  4. Use npm run test:ci in your CI (the key things are that coverage is enabled and that the --ci argument is present)

Going Forward

Concurrency and Parallelism

If you're leveraging parallelism to do test splitting and running your tests concurrently on CI (e.g. fan-out/fan-in), a few adjustments to the pattern are needed.

<img src="https://circleci.com/docs/assets/img/docs/fan-out-in.png" width="300">
  1. Remove the posttest:ci script - you'll need to explicitly invoke coverage validation as a separate step after you gather coverage on the concurrent runs.

Use jest to generate the files to be tested so you ensure you have parity with the test run and coverage gathering used to generate the exceptions:

TESTFILES=$(npx jest --listTests | sed s:$PWD/:: | circleci tests split --split-by=timings --show-counts)
npm run test:ci $TESTFILES
  1. You will need to configure your CI to keep the full json coverage reports around for a follow-up validation step in your workflow. Ensure these can be located later under the coverage output directory (both jest and mergeCoveragePath should be set to the same directory). For CircleCI, this means adding them to a workspace folder with unique names:
// example
COVERAGE_REPORT_SHARD=coverage/coverage-final${CIRCLE_NODE_INDEX}.json
npm run test:ci $TESTFILES && mv coverage/coverage-final.json $COVERAGE_REPORT_SHARD
  1. Setup an additional job in the CI (e.g. test_coverage) that runs after the concurrent testing is completed.
    • Explicitly run test:validateCoverage with the merge argument: npm run test:validateCoverage -- --merge.

Example config.json (the mergeCoveragePath directory should match jest):

{
  ...
  "mergeCoveragePath": "coverage",
  ...
}

CLI

$ jest-a-coverage-slip-detector --help
Usage: jest-a-coverage-slip-detector [options]

Options:
  --help, -h             Show this help

  --update               Update exceptions with improved coverage levels.
                         Used to 'ratchet' up coverage after improving it.

  --force-update         Record current coverage errors as exceptions.
                         Used to:
                           - Snapshot current coverage errors as legacy exceptions.
                           - Force accept a reduction in coverage.

  --merge                Merges together concurrently collected coverage

  --report-only          Exit successfully even if coverage errors are detected.

Testing locally

FAQ

After I'm setup with this library, what if I decide to raise the coverage goal higher for new code?

Do I need to use different test commands on dev than I would on CI?

Why do I only see the coverage errors on CI and not locally?

What if I'm running tests locally, will I be slowed down by coverage scanning?

How do I incrementally add test coverage to a previously uncovered file without having testing fail due to the goal being unmet?

What exactly is the purpose of withJestSlipDetection?