Home

Awesome

Jenkins Coverage Plugin

Join the chat at Gitter/Matrix Jenkins Plugin Installs Jenkins GitHub Actions Codecov CodeQL

The Jenkins Coverage Plug-in collects reports of code coverage or mutation coverage tools. It has support for the following report formats:

If your coverage tool is not yet supported by the coverage plugin, feel free to provide a pull request for the Coverage Model.

The plugin publishes a report of the code and mutation coverage in your build, so you can navigate to a summary report from the main build page. Additionally, the plugin gathers several metrics (lines of code, cyclomatic complexity, number of tests per class) and visualizes these results along with the coverage information. From there you can also dive into the details:

The initial version of this plugin has been developed by Shenyu Zheng in GSoC 2018. After several incompatible improvements of the pipeline steps and API classes, we decided to move the whole source code to a clean and new plugin. The old plugin containing the deprecated steps and code is still available at GitHub and Jenkins.

This code of this plugin is also available as a standalone GitHub or GitLab action that runs without Jenkins now:

Quality Monitor GitHub Action

Features

The code coverage plug-in provides the following features when added as a post-build action (or step) to a job:

Coverage overview and trend

Usage

:exclamation: The plugin does not run the code coverage, it just visualizes the results reported by such tools. You still need to enable and configure the code coverage tool in your build file or Jenkinsfile. :exclamation:

Supported parsers

The Coverage Plug-in supports the following report formats:

Some of these report files are generated by other tools and may contain invalid or inconsistent information. By default, the plugin tries to fail fast if such a broken file is detected. You can disable this behavior by setting the property ignoreParsingErrors to true. In this case, the plugin will try to parse as much information as possible from the report file.

Supported project types

The Coverage Plug-in supports the following Jenkins project types:

Freestyle project

Enable the "Record code coverage results" publisher in the Post-build Actions section of your job. Select at least one coverage tool and specify the path to the report file. If you do not specify a path, the plugin will search for the report file in the workspace using the default pattern of the tool.

Pipeline example

The recording step also supports pipeline configuration (scripted or declarative). The simplest way to generate the corresponding pipeline code is by using Jenkins' Snippet Generator: there you can configure all available properties of the step by using a rich user interface with inline help and validation. A sample step definition is given in the following code snippet. This step records coverage results of JaCoCo with the default file name pattern. The coverage information is rendered with the source code for every build, and quality gates have been set up for line and branch coverages. I.e., if the line or branch coverage does not reach the threshold of 60%, then the build will be marked as unstable.

recordCoverage(tools: [[parser: 'JACOCO']],
        id: 'jacoco', name: 'JaCoCo Coverage',
        sourceCodeRetention: 'EVERY_BUILD',
        qualityGates: [
                [threshold: 60.0, metric: 'LINE', baseline: 'PROJECT', unstable: true],
                [threshold: 60.0, metric: 'BRANCH', baseline: 'PROJECT', unstable: true]])

Reference build (baseline)

One unique feature of the Coverage plugin is the delta computation of coverage metrics with respect to a baseline build (reference build). The plugin reads the results of the reference build and compares these results with the current results: for each coverage metric, a delta value is computed. The delta values are shown as absolute coverages of the project, the coverages of the modified files, or the coverages of the modified lines. This helps to see how the code changes actually affect the code coverage.

In order to compute this classification, the plugin requires a reference build (baseline). When selecting a baseline, we need to distinguish two different use cases, which are documented in the next sections.

Selecting a baseline from the current job

When a team wants to investigate how the coverage of the project changes over the time, we need to simply look back in the history of the same Jenkins job and select another build that we can use to compare the results with. Such a Jenkins job typically builds the main branch of the source control system. This can be achieved by using discoverReferenceBuild before the step to record the code coverage:

discoverReferenceBuild()
recordCoverage(tools: [[parser: 'JACOCO']])

Selecting a baseline in the target job

:warning: This feature requires the installation of an additional plugin: Git Forensics Plugin.

For more complex branch source projects (i.e., projects that build several branches and pull requests in a connected job hierarchy) it makes more sense to select a reference build from a job that builds the actual target branch (i.e., the branch the current changes will be merged into). Here one typically is interested what changed in the branch or pull request over the main branch (or any other target branch). That means we want to see how the coverage changes when new code will be submitted by a branch or pull request.

If you are using a Git branch source project, the Jenkins job that builds the target branch will be selected automatically by running the reference recorder step. Simply call the step discoverGitReferenceBuild before the step to record the code coverage:

discoverGitReferenceBuild()
recordCoverage(tools: [[parser: 'JACOCO']])

Selecting the correct reference build is not that easy as it looks, since the main branch of a project will evolve more frequently than a specific feature or bugfix branch. That means if we want to compare the results of a pull request with the results of the main branch we need to select a build from the main branch that contains only commits that are also part of the branch of the associated pull request.

Therefore, the Git Forensics plugin automatically tracks all commits of a Jenkins build and uses this information to identify a build in the target branch that matches best with the commits in the current branch. Please have a look at the documentation of the Git Forensics plugin to see how this is achieved in detail.

This algorithm can be used for plain Git SCM freestyle projects or pipelines as well. In this case, we cannot get the target branch information automatically from the Git branch source API. Therefore, you need to manually specify the Jenkins job that builds the target branch in the parameter referenceJob. See the following sample pipeline snippet for an example on how to discover a baseline from such a reference job:

discoverGitReferenceBuild referenceJob: 'my-reference-job'
recordCoverage(tools: [[parser: 'JACOCO']])

Rendering of the source code

The plugin will automatically find your source code files to create a report that shows the source code in combination with the achieved code coverage results. You can change the strategy that should be used to store the colored source code files with the property sourceCodeRetention. If your server has not enough free space available to store the sources for all builds it might make sense to store only the coverage results of the last build. In this case, the plugin will automatically discard old results before the new sources will be stored. Or, if you do not need the source files at all, you can deactivate the storing of source code files. The following options are supported:

For Java projects, the source code rendering typically works out-of-the-box since the coverage tools export the results into a report that contains the exact locations of the source code files (absolute path). If this automatic detection does not work in your case, then you can specify a path prefix to the sources by using the option sourceDirectories. This property can be filled with one or more relative paths within the workspace that should be searched for the source code. You can also specify absolute paths, but then you need to make sure that those paths are approved by an administrator in the configuration section of the Prism plugin in Jenkins´ global configuration. The following example shows how to specify such a path prefix:

recordCoverage(tools: [[parser: 'JACOCO']], 
            sourceCodeRetention: 'MODIFIED', 
            sourceDirectories: [[path: 'plugin/src/main/java']])

Token macro support

The coverage plugin provides the token COVERAGE that could be used in additional post build processing steps, e.g. in the mailer. In order to use this token you need to install the Token Macro plugin. The token has the following optional parameters:

Examples:

Remote API

We provide a remote API to retrieve a coverage overview, using the following URL: https://[jenkins-url]/job/[job-name]/[build-number]/coverage/api/json?pretty=true.

Example output:

{
  "_class" : "io.jenkins.plugins.coverage.metrics.restapi.CoverageApi",
  "modifiedFilesDelta" : {
    "branch" : "+1.72%",
    "class" : "-3.54%",
    "complexity" : "-236",
    "complexity-density" : "+0.47%",
    "file" : "+0.00%",
    "instruction" : "+0.16%",
    "line" : "-0.48%",
    "loc" : "-482",
    "method" : "+1.23%",
    "module" : "+0.00%",
    "package" : "+0.00%"
  },
  "modifiedFilesStatistics" : {
    "branch" : "83.91%",
    "class" : "93.33%",
    "complexity" : "392",
    "complexity-density" : "+50.19%",
    "file" : "100.00%",
    "instruction" : "88.19%",
    "line" : "87.96%",
    "loc" : "781",
    "method" : "86.18%",
    "module" : "100.00%",
    "package" : "100.00%"
  },
  "modifiedLinesDelta" : {
    "branch" : "+8.95%",
    "file" : "+0.00%",
    "line" : "+3.85%",
    "loc" : "-610",
    "module" : "+0.00%",
    "package" : "+0.00%"
  },
  "modifiedLinesStatistics" : {
    "branch" : "92.86%",
    "file" : "100.00%",
    "line" : "91.81%",
    "loc" : "171",
    "module" : "100.00%",
    "package" : "100.00%"
  },
  "projectDelta" : {
    "branch" : "+4.43%",
    "class" : "+2.94%",
    "complexity" : "-8",
    "complexity-density" : "+1.28%",
    "file" : "+4.00%",
    "instruction" : "+2.59%",
    "line" : "+3.37%",
    "loc" : "-50",
    "method" : "+1.28%",
    "module" : "+0.00%",
    "package" : "+0.00%"
  },
  "projectStatistics" : {
    "branch" : "82.19%",
    "class" : "96.88%",
    "complexity" : "628",
    "complexity-density" : "+49.72%",
    "file" : "100.00%",
    "instruction" : "88.03%",
    "line" : "88.44%",
    "loc" : "1263",
    "method" : "84.94%",
    "module" : "100.00%",
    "package" : "100.00%"
  },
  "qualityGates" : {
    "overallResult" : "SUCCESS",
    "resultItems" : [
      {
        "qualityGate" : "Overall project - Line Coverage",
        "result" : "SUCCESS",
        "threshold" : 60.0,
        "value" : "88.44%"
      },
      {
        "qualityGate" : "Overall project - Branch Coverage",
        "result" : "SUCCESS",
        "threshold" : 60.0,
        "value" : "82.19%"
      }
    ]
  },
  "referenceBuild" : "<a href=\"http://localhost:8080/job/coverage-model-history/10/\" class=\"model-link inside\">coverage-model-history #10</a>"
}

More concrete, the line coverage for modified code lines is provided per modified file, using the following URL: https://[jenkins-url]/job/[job-name]/[build-number]/coverage/modified/api/json?pretty=true.

Example output:

{
  "_class": "io.jenkins.plugins.coverage.metrics.restapi.ModifiedLinesCoverageApi",
  "files": [
    {
      "fullyQualifiedFileName": "io/jenkins/plugins/coverage/metrics/restapi/ModifiedLinesCoverageApi.java",
      "modifiedLinesBlocks": [
        {
          "startLine": 30,
          "endLine": 35,
          "type": "MISSED"
        }
      ]
    },
    {
      "fullyQualifiedFileName": "io/jenkins/plugins/coverage/metrics/restapi/ModifiedLinesBlocks.java",
      "modifiedLinesBlocks": [
        {
          "startLine": 80,
          "endLine": 81,
          "type": "COVERED"
        }
      ]
    }
  ]
}