Awesome

vcflib

A C++ library for parsing and manipulating VCF files.

Vcflib and related tools are the workhorses in bioinformatics for processing the VCF variant calling format. See

A spectrum of free software tools for processing the VCF variant call format: vcflib, bio-vcf, cyvcf2, hts-nim and slivar. Garrison E, Kronenberg ZN, Dawson ET, Pedersen BS, Prins P (2022), PLoS Comput Biol 18(5): e1009123. https://doi.org/10.1371/journal.pcbi.1009123

(bibtex reference at the end of this page)

news

January 2023: this is vcflib's first Humpty Dumpty release: vcfcreatemulti is the natural companion to vcfwave. Often variant callers are not perfect. vcfwave with its companion tool vcfcreatemulti can take an existing VCF file that contains multiple complex overlapping and even nested alleles and, unlike Humpty Dumpty, take them apart and put them together again. Thereby, hopefully, creating sane VCF output that is useful for analysis and getting rid of false positives.

We created these tools by including the state-of-the-art biWFA wavefront aligner. The tools are particularly useful for the output from structural variation callers and pangenome genotypers, such as used by the Human Pangenome Reference Consortium (HPRC) because of overlapping ALT segments.

See also RELEASE_NOTES.md

May 2022: the vcflib paper has been published on PLoS Computational Biology!

April 2022: vcflib has just gone pangenome!

By introducing the wavefront algorithm we can now realign long sequences and reduce call complexity (and FPs!) introduced by pangenome variant callers using the new vcfwave tool.

See also RELEASE_NOTES.md

overview

The Variant Call Format (VCF) is a flat-file, tab-delimited textual format that describes reference-indexed variations between individuals. VCF provides a common interchange format for the description of variation in individuals and populations of samples, and has become the de facto standard reporting format for a wide array of genomic variant detectors.

vcflib provides methods to manipulate and interpret sequence variation described by VCF. It is both:

• an API for parsing and operating on records of genomic variation as it can be described by the VCF format
• a collection of command-line utilities for executing complex manipulations on VCF files

vclib is both a library (with an API) and a collection of useful tools. The API provides a quick and extremely permissive method to read and write VCF files. Extensions and applications of the library provided in the included utilities (*.cpp) comprise the vast bulk of the library's utility.

We have also added infrastructure to write Python bindings. See below.

Short index:

• Install
• Usage
• TOOLS
  • Filter
  • Metrics
  • Phenotype
  • Genotype
  • Transformation
  • Statistics
  • Scripts
  • Python bindings
• Link library
• Build from source
• Development
• LICENSE
• Credit work
  • Bibtex reference

INSTALL

For latest updates see RELEASE NOTES.

Bioconda

Conda installs in user land without root access

conda install -c bioconda vcflib

Homebrew

Homebrew installs on Linux and Mac OSX

brew install brewsci/bio/vcflib

Debian

For Debian and Ubuntu

apt-get install libvcflib-tools libvcflib-dev

GNU Guix

We develop against guix and vcflib is packaged as

guix package -i vcflib

See also the Guix shell below.

USAGE

Users are encouraged to drive the utilities in the library in a streaming fashion, using Unix pipes to fully utilize resources on multi-core systems. Piping provides a convenient method to interface with other libraries (vcf-tools, BedTools, GATK, htslib, bio-vcf, bcftools, freebayes) which interface via VCF files, allowing the composition of an immense variety of processing functions. Examples can be found in the scripts, e.g. ./scripts.

TOOLS

filter

metrics

phenotype

genotype

transformation

statistics

See also vcflib.md.

scripts

The vcflib source repository contains a number of additional scripts. Click on the link to see the source code.

python

vcflib has rudimentary python bindings, but the are easy to build up on. See pyvcflib.

Development

build from source

VCFLIB uses the cmake build system, after a recursive checkout of the sources make the files in the 'build' directory with:

git clone --recursive https://github.com/vcflib/vcflib.git
cd vcflib
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DZIG=OFF -DOPENMP=OFF ..
cmake --build .
cmake --install .

and to run the tests

ctest . --verbose

Executables are built into the ./build directory in the repository.

Note, if you have an existing repo update submodules with

git submodule update --init --recursive --progress
cd build
cmake --build . --target clean

The most interesting one, perhaps, is wfa2lib. Bundling is no longer the default, but it is a submodule that will install an optimized libwfa2.so. Set up compilation with

cmake -DCMAKE_BUILD_TYPE=Release -DWFA_GITMODULE=ON  ..
cmake --build .
ctest .
cmake --install .

Build dependencies can be viewed in the github-CI scripts (see badges above), as well as guix.scm used by us to create the build environment (for instructions see the header of guix.scm). Essentially:

• cmake
• C++ compiler
• htslib
• tabixpp
• WFA2 (wfa2lib)
• pybind11 (for testing)

For include files add

• libhts-dev
• libtabixpp-dev
• libtabixpp0

And for some of the VCF executables

• zig 0.13 (disable with cmake -DZIG=OFF)
• python
• perl

To use Zig compiled tools, such as vcfcreatemulti, make sure a recent zig compiler is in the PATH. The zig dependency is for the recently upgraded vcfcreatemulti that can be run after vcfwave to combine overlapping records. It is unfortunate that zig is not part of some large Linux distributions.

Note you need to add the full path before running cmake and make. From the build dir this should work:

cd build
export PATH=$(pwd)/../zig-linux-x86_64-0.13.0:$PATH
zig version
cmake  -DCMAKE_BUILD_TYPE=Debug -DWFA_GITMODULE=1 ..
make VERBOSE=1

More on zig can be found in the souce code README.

Using a recent C++ compiler

You may encounter something like libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ...vcflib/build/pyvcflib.cpython-310-x86_64-linux-gnu.so). This is caused by the python pyvcflib not running against a python interpreter compiled with the same libstdc++ dependency. That is a bit nasty to solve. For now make sure to match C++ compilers for the python module.

Using a different htslib

Check out htslib in tabixpp (recursively) and

cmake -DHTSLIB_LOCAL:STRING=./htslib/ ..
cmake --build .

link library

The standard build creates build/vcflib.a. Take a hint from the cmake file that builds all the vcflib tools.

distro builds

Distros, such as Debian, should build with something like

cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DZIG=OFF ..

See the CMakeLists.txt header for more.

source code

See vcfecho.cpp for basic usage. Variant.h and Variant.cpp describe methods available in the API. vcflib is incorporated into several projects, such as freebayes, which may provide a point of reference for prospective developers. Note vcflib contains submodules (git repositories) comprising some dependencies. A full Guix development environment we use is defined here.

adding tests

vcflib uses different test systems. The most important one is the doctest because it doubles as documentation. For an example see vcf2tsv.md which can be run from the command line with

cd test
python3 -m doctest -o NORMALIZE_WHITESPACE -o REPORT_UDIFF pytest/vcf2tsv.md

We also added support for python bindings and unit tests. See realign.py for an example.

Support

The developers are on the vcflib matrix channel. Please do not use the github issue tracker for support issues!

Contributing

To contribute code to vcflib send a github pull request. We may ask you to add a working test case as described in 'adding tests'.

LICENSE

This software is distributed under the free software MIT LICENSE.

CREDIT

Citations are the bread and butter of Science. If you are using this software in your research and want to support our future work, please cite the following publication:

Please cite:

A spectrum of free software tools for processing the VCF variant call format: vcflib, bio-vcf, cyvcf2, hts-nim and slivar. Garrison E, Kronenberg ZN, Dawson ET, Pedersen BS, Prins P (2022), PLoS Comput Biol 18(5): e1009123. https://doi.org/10.1371/journal.pcbi.1009123

Bibtex reference

@article{10.1371/journal.pcbi.1009123,
    doi = {10.1371/journal.pcbi.1009123},
    author = {Garrison, Erik AND Kronenberg, Zev N. AND Dawson, Eric T. AND Pedersen, Brent S. AND Prins, Pjotr},
    journal = {PLOS Computational Biology},
    publisher = {Public Library of Science},
    title = {A spectrum of free software tools for processing the VCF variant call format: vcflib, bio-vcf, cyvcf2, hts-nim and slivar},
    year = {2022},
    month = {05},
    volume = {18},
    url = {https://doi.org/10.1371/journal.pcbi.1009123},
    pages = {1-15}
}

Below the prepublished version of our paper

@article {Garrison2021.05.21.445151,
	author = {Garrison, Erik and Kronenberg, Zev N. and Dawson, Eric T. and Pedersen, Brent S. and Prins, Pjotr},
	title = {Vcflib and tools for processing the VCF variant call format},
	elocation-id = {2021.05.21.445151},
	year = {2021},
	doi = {10.1101/2021.05.21.445151},
	publisher = {Cold Spring Harbor Laboratory},
	URL = {https://www.biorxiv.org/content/early/2021/05/23/2021.05.21.445151},
	eprint = {https://www.biorxiv.org/content/early/2021/05/23/2021.05.21.445151.full.pdf},
	journal = {bioRxiv}
}