Home

Awesome

code-docstring-corpus

This repository contains preprocessed Python functions and docstrings for automated code documentation (code2doc) and automated code generation (doc2code) tasks.

Paper: https://arxiv.org/abs/1707.02275

Update

The code-docstring-corpus version 2, with class declarations, class methods, module docstrings and commit SHAs is now available in the directory V2

Installation

The dependencies can be installed using pip:

pip install -r requirements.txt

Extraction scripts require AST Unparser ( https://github.com/simonpercivall/astunparse ), NMT tokenization requires the Moses tokenizer scripts ( https://github.com/moses-smt/mosesdecoder )

Details

We release a parallel corpus of 150370 triples of function declarations, function docstrings and function bodies. We include multiple corpus splits, and an additional "monolingual" code-only corpus with corresponding synthetically generated docstrings.

The corpora were assembled by scraping from open source GitHub repository with the GitHub scraper used by Bhoopchand et al. (2016) "Learning Python Code Suggestion with a Sparse Pointer Network" (paper: https://arxiv.org/abs/1611.08307 - code: https://github.com/uclmr/pycodesuggest ) .

The Python code was then preprocessed to normalize the syntax, extract top-level functions, remove comments and semantically irrelevant whitespaces, and separate declarations, docstrings (if present) and bodies. We did not extract classes and their methods.

directorydescription
parallel-corpusMain parallel corpus with a canonical split in 109108 training triples, 2000 validation triples and 2000 test triples. Each triple is annotated by metadata (repository owner, repository name, source file and line number). Also two versions of the above corpus reassembled into pairs: (declaration+body, docstring) and (declaration+docstring, body), for code documentation tasks and code generation tasks, respectively. You may refer to the Readme in this folder for descriptions about escape tokens
code-only-corpusA code-only corpus of 161630 pairs of function declarations and function bodies, annotated with metadata.
backtranslations-corpusA corpus of docstrings automatically generated from the code-only corpus using Neural Machine Translation, to enable data augmentation by "backtranslation"
nmt-outputsTest and validation outputs of the baseline Neural Machine Translation models.
repo_split.parallel-corpusAn alternate train/validation/test split of the parallel corpus which is "repository-consistent": no repository is split between training, validation or test sets.
repo_split.code-only-corpusA "repository-consistent" filtered version of the code-only corpus: it only contains fragments which appear in the training set of the above repository.
scriptsPreprocessing scripts used to generate the corpora.
V2code-docstring-corpus version 2, with class declarations, class methods, module docstrings and commit SHAs.

Baseline results

In order to compute baseline results, the data from the canonical split (parallel-corpus directory) was further sub-tokenized using Sennrich et al. (2016) "Byte Pair Encoding" (paper: https://arxiv.org/abs/1508.07909 - code: https://github.com/rsennrich/subword-nmt ). Finally, we trained baseline Neural Machine Translation models for both the code2doc and the doc2code tasks using Nematus (Sennrich et al. 2017, paper: https://arxiv.org/abs/1703.04357 - code: https://github.com/rsennrich/nematus ).

Baseline outputs are available in the nmt-outputs directory.

We also used the code2doc model to generate the docstring corpus from the code-only corpus which is available in the backtranslations-corpus directory.

ModelValidation BLEUTest BLEU
declbodies2desc.baseline14.0313.84
decldesc2bodies.baseline10.3210.24
decldesc2bodies.backtransl10.8510.90

Bleu scores are computed using Moses multi-bleu.perl script

Reference

If you use this corpus for a scientific publication, please cite: Miceli Barone, A. V. and Sennrich, R., 2017 "A parallel corpus of Python functions and documentation strings for automated code documentation and code generation" arXiv:1707.02275 https://arxiv.org/abs/1707.02275