Awesome

BLUE Benchmark with Transformers

***** New May 14th, 2020: ouBioBERT (full) is released *****
***** New April 15th, 2020: released *****

Thank you for your interest in our research!
The biomedical language understanding evaluation (BLUE) benchmark is a collection of resources for evaluating and analyzing biomedical natural language representation models (Peng et al., 2019).
This repository provides our implementation of fine-tuning for the BLUE benchmark with 🤗/Transformers.
Our demonstration models are available now.

Preparations

1. Download the benchmark dataset from https://github.com/ncbi-nlp/BLUE_Benchmark
2. Save pre-trained models to your directory. For example, BioBERT, clinicalBERT, SciBERT, BlueBERT and so on.
3. Try to use our code in utils. Examples of the command can be found in scripts.

Tips

If you download Tensorflow models, converting them into PyTorch ones comforts your fine-tuning.
Converting Tensorflow Checkpoints

export BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12

transformers-cli convert --model_type bert \
  --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
  --config $BERT_BASE_DIR/bert_config.json \
  --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin

Our models

• Demonstration models for our research
  • ouBioBERT-Base, Uncased # 20200514 (recommended)
    • the best score on the BLUE benchmark
    • trained on Focused PubMed abstracts with the Other PubMed abstracts.
    • max_seq_length=512
    • also community-uploaded on Hugging Face, refer to https://huggingface.co/models.
    • the sample command of the fine-tuning is here.
  • ouBioBERT-Base, Uncased (demo) # 20200415
    • trained on Focused PubMed abstracts with the Other PubMed abstracts.
    • max_seq_length=128
  • BERT (sP + B + enW) # 20200512
    • a validation model for our method.
    • trained on a small biomedical corpus with BooksCorpus and Wikipedia.

Table: List of the text corpora used for our models.

• Small PubMed abstracts (sP): extracted from PubMed baseline more associated with clinical research and translational research of human disease by using each MeSH IDs.
• Focused PubMed abstracts (fP): articles more related to human beings.
• Other PubMed abstracts (oP): articles other than Focused PubMed abstracts.

Results

Table: BLUE scores of BERT (sP + B + W) compared with those of all the BERT-Base variants for the biomedical domain as of April 2020.
Bold indicates the best result of all.

Table: Performance of ouBioBERT on the BLUE task.
The numbers are mean (standard deviation) on five different random seeds.
The best scores are in bold.

Table of Contents

• Preparations
• Our models
• Results
• BLUE Tasks
• Sentence similarity
  • MedSTS
  • BIOSSES
• Named-entity recognition
  • BC5CDR-disease
  • BC5CDR-chemical
  • ShARe/CLEFE
• Relation extraction
  • DDI
  • ChemProt
  • i2b2 2010
• Document multilabel classification
  • HoC
• Inference task
  • MedNLI
• Total score
• Citing
• Funding
• Acknowledgments
• References

BLUE Tasks

Sentence similarity

• The sentence-similarity task is to predict similarity scores on the basis of sentence pairs.
• Metrics: Pearson correlation coefficients
  • We use scipy.stats.pearsonr().

MedSTS

MedSTS is a corpus of sentence pairs selected from the clinical data warehouse of Mayo Clinic and was used in the BioCreative/OHNLP Challenge 2018 Task 2 as ClinicalSTS (Wang et al., 2018).
Please visit the website or contact to the 1st author to obtain a copy of the dataset.

BIOSSES

BIOSSES is a corpus of sentence pairs selected from the Biomedical Summarization Track Training Dataset in the biomedical domain (Soğancıoğlu et al., 2017).

Known problems

The BIOSSES dataset is very small, therefore it causes unstable performance of fine-tuning.

Named-entity recognition

• The aim of the Named-entity recognition task is to predict mention spans given in a text.
• Metrics: strict version of F1-score (exact phrase matching).
  • We use a primitive approach descirbed below to deal with disjoint mentions.

Known problems

There are some irregular patterns:

• starting with I: caused by long phrases split in the middle (example).
• I next to O: due to discontinuous mentions. It is often observed in ShARe/CLEFE (example).

conlleval.py appears to count them as different phrases.
Then, we manage this problem by the following method on evaluation:

1. Example:

2. skip blank line and concat all the tags in the sentence into a one-dimensional array.

3. get the token index of phrases that start with B.

• y_true: 2_3, 7_8_10, 13_14_15_17_18
• y_pred: 2_3, 7_8, 13_14_15_17_19

4. calculate metrics: utils/metrics/ner.py

y_true = set(y_true)
y_pred = set(y_pred)

TP = len(y_true & y_pred)           # 1: {2_3}
FN = len(y_true) - TP               # 2: {7_8_10, 13_14_15_17_18}
FP = len(y_pred) - TP               # 2: {7_8, 13_14_15_17_19}
prec = TP / (TP + FP)               # 1 / (1 + 2) = 0.33
rec = TP / (TP + FN)                # 1 / (1 + 2) = 0.33
fb1 = 2 * rec * prec / (rec + prec) # = 0.33

BC5CDR-disease

BC5CDR is a collection of 1,500 PubMed titles and abstracts selected from the CTD-Pfizer corpus and was used in the BioCreative V chemical-disease relation task (Li et al., 2016).

BC5CDR-chemical

An example of starting with I: test.tsv#L78550-L78598<a id="startingwithi"></a>

  
Compound	10510854	553	O  
7e	-	562	O  
,	-	564	O  
5	-	566	B  
-	-	567	I  
{	-	568	I  
2	-	569	I  
-	-	570	I  
// -------------
1H	-	637	I  
-	-	639	I  
  
indol	10510854	641	I  
-	-	646	I  
2	-	647	I  
-	-	648	I  
one	-	649	I  
,	-	652	O  
// -------------

ShARe/CLEFE

ShARe/CLEFE eHealth Task 1 Corpus is a collection of 299 clinical free-text notes from the MIMIC II database (Suominen et al.,2013).
Please visit the website and sign up to obtain a copy of the dataset.
An example of I next to O: Test.tsv#L112-L118<a id="inexttoo"></a>
You'd better check out these original files, too:

• Task1Gold_SN2012/Gold_SN2012/00176-102920-ECHO_REPORT.txt#L2
• Task1TestSetCorpus100/ALLREPORTS/00176-102920-ECHO_REPORT.txt#L21

The	00176-102920-ECHO_REPORT	426	O
left	-	430	B
atrium	-	435	I
is	-	442	O
moderately	-	445	O
dilated	-	456	I
.	-	463	O

Relation extraction

• The aim of the relation-extraction task is to predict relations and their types between the two entities mentioned in the sentences. The relations with types were compared to annotated data.
• Following the implimentation of the BLUE benchmark, we treated the relation extraction task as a sentence classification by replacing two named entity mentions of interest in the sentence with predefined tags (Lee et al., 2019).
  • ORIGINAL: Citalopram protected against the RTI-76-induced inhibition of SERT binding.
  • REPLACED: @CHEMICAL$ protected against the RTI-76-induced inhibition of @GENE$ binding.
  • RELATION: citalopram and SERT has a chemical-gene relation.
• Evaluation:
  1. predict classes containing "false".
  2. aggregate TP, FN, and FP in each class.
  3. calculate metrics excluding the "false" class.
• Metrics: micro-average F1-score.
  • We use sklearn.metrics.confusion_matrix() and compute TP, FP, FN and TN on each class.

DDI

DDI extraction 2013 corpus is a collection of 792 texts selected from the DrugBank database and other 233 Medline abstracts (Herrero-Zazo et al., 2013).

ChemProt

ChemProt comprises 1,820 PubMed abstracts with chemical–protein interactions and was used in the BioCreative VI text mining chemical–protein interac-tions shared task (Krallinger et al, 2017).

i2b2 2010

i2b2 2010 shared task collection comprises 170 documents for training and 256 for testing (Uzuner et al., 2011).

Known problems

The development dataset is very small, then it is difficult to determine the best model.

Document multilabel classification

• The multilabel-classification task predicts multiple labels from the texts.

HoC

Labels: (IM) Activating invasion & metastasis, (ID) Avoiding immune destruction, (CE) Deregulating cellular energetics, (RI) Enabling replicative immortality, (GS) Evading growth suppressors, (GI) Genome instability & mutation, (A) Inducing angiogenesis, (CD) Resisting cell death, (PS) Sustaining proliferative signaling, (TPI) tumor promoting inflammation
Note: This table shows the number of each label on the sentence level, rather than on the abstract level.

• Train: sentences: 10527/ articles: 1108
• Dev: sentences: 1496/ articles: 157
• Test: sentences: 2896/ articles: 315

HoC (the Hallmarks of Cancers corpus) comprises 1,580 PubMed publication abstracts manually annotated using ten currently known hallmarks of cancer (Baker et al., 2016).

• Evaluation:
  1. predict multi-labels for each sentence in the document.
  2. combine the labels in one document and compare them with the gold-standard.
• Metrics: example-based F1-score on the abstract level (Zhang and Zhou, 2014; Du et al., 2019).
  • We use eval_hoc.py from the BLUE benchmark repository to calculate the metrics.

Inference task

• The inference task aims to predict whether the relationship between the premise and hypothesis sentences is contradiction, entailment, or neutral.
• Metrics: overall accuracy
  • We use sklearn.metrics.confusion_matrix() and sklearn.metrics.accuracy_score() to compute TP, FP, FN and TN on each class and overall accuracy.

MedNLI

MedNLI is a collection of sentence pairs selected from MIMIC-III (Romanov and Shivade, 2018).
Please visit the website and sign up to obtain a copy of the dataset.

Total score

Following the practice in Peng et al. (2019), we use a macro-average of Pearson scores and F1-scores to determine a pre-trained model's position.
The results are above.

Citing

If you use our work in your research, please kindly cite the following papers:
the original paper of the BLUE Benchmark

• Peng Y, Yan S, Lu Z. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019). 2019:58-65.

Our research

@misc{2005.07202,
Author = {Shoya Wada and Toshihiro Takeda and Shiro Manabe and Shozo Konishi and Jun Kamohara and Yasushi Matsumura},
Title = {A pre-training technique to localize medical BERT and enhance BioBERT},
Year = {2020},
Eprint = {arXiv:2005.07202},
}

Funding

This work was supported by Council for Science, Technology and Innovation (CSTI), cross-ministerial Strategic Innovation Promotion Program (SIP), "Innovative AI Hospital System" (Funding Agency: National Institute of Biomedical Innovation, Health and Nutrition (NIBIOHN)).

Acknowledgments

We are grateful to the authors of BERT to make the data and codes publicly available. We thank the NVIDIA team because their implementation of BERT for PyTorch enabled us to pre-train BERT models on our local machine. We would also like to thank Yifan Peng and shared-task organizers for publishing the BLUE benchmark.

References

• <a id="ypeng"></a>Peng Y, Yan S, Lu Z. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In Proceedings of the Workshop on Biomedical Natural Language Processing (BioNLP). 2019: 58-65.
• <a id="ywang"></a>Wang Y, Afzal N, Fu S, Wang L, Shen F, Rastegar-Mojarad M, Liu H. MedSTS: a resource for clinical semantic textual similarity. Language Resources and Evaluation. 2018 Jan 1:1-6.
• <a id="gsogancioglu"></a>Soğancıoğlu G, Öztürk H, Özgü A. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics. 2017 Jul 15; 33(14): i49–i58.
• <a id="jli"></a>Li J, Sun Y, Johnson RJ, Sciaky D, Wei CH, Leaman R, Davis AP, et al.. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database: the journal of biological databases and curation. 2016.
• <a id="hsuominen"></a>Suominen H, Salanterä S, Velupillai S, Chapman WW, Savova G, Elhadad N, Pradhan S, et al.. Overview of the ShARe/CLEF eHealth evaluation lab 2013. Information Access Evaluation Multilinguality, Multimodality, and Visualization. 2013. Springer. 212–231.
• <a id="jlee"></a>Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. 2019;36(4):1234-40.
• <a id="hzazo"></a>Herrero-Zazo M, Segura-Bedmar I, Martínez P, Declerck T. The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. Journal of biomedical informatics. 2013 46: 914–920.
• <a id="mkrallinger"></a>Krallinger M, Rabal O, Akhondi SA, Pérez MP, Santamaría JL, Rodríguez GP, Tsatsaronis G, et al.. Overview of the BioCreative VI chemical-protein interaction track. In Proceedings of BioCreative. 2017. 141–146.
• <a id="ouzuner"></a>Uzuner Ö,South BR,Shen S, DuVall SL. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association (JAMIA). 2011 18: 552–556.
• <a id="sbaker"></a>Baker S, Silins I, Guo Y, Ali I, Högberg J, Stenius U, Korhonen A. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics (Oxford, England). 2016 32: 432–440.
• <a id="zz"></a>Zhang ML, Zhou ZH. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering. 2014 26(8): 1819–1837.
• <a id="jdu"></a>Du J, Chen Q, Peng Y, Xiang Y, Tao C, Lu Z. ML-Net: multilabel classification of biomedical texts with deep neural networks. Journal of the American Medical Informatics Association (JAMIA). 2019 Nov; 26(11); 1279–1285.
• <a id="aromanov"></a>Romanov A, Shivade C. Lessons from Natural Language Inference in the Clinical Domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 1586-1596.