Home

Awesome

DataSeer-ML

License Docker Hub SWH

Logo DataSeer

dataseer-ml is a GROBID module aiming at identifying implicit mentions of datasets in a scientific article. These identified datasets are further classified in a hierarchy of dataset types, these data types being directly derived from MeSH. It is a back-end service used by the DataSeer-Web application. Most of the datasets discussed in scientific articles are actually not named, but these data are part of the disclosed scientific work and should be shared properly to meet the FAIR requirements.

The goal of this process is to further drive the authors of the article to the best research data sharing practices, i.e. to ensure that the dataset is associated with data availability statement, permanent identifiers and in general requirements regarding Open Science and reproducibility. This further process is realized by the dataseer web application which includes a GUI to be used by the authors, suggesting data sharing policies based on the predicted data types for each identified dataset.

The module can process a variety of scientific article formats, including mainstream publisher's native XML submission formats: PDF, TEI, JATS/NLM, ScholarOne, BMJ, Elsevier staging format, OUP, PNAS, RSC, Sage, Wiley, etc.

.docx format is also supported in a GROBID specific branch, but not yet merged.

Approach

Dataset identification and classification

The processing of an article follows 5 steps:

  1. Given an article to be processed by DataSeer:

1.1. if the format is PDF or docx, the document is first parsed and structured automatically by Grobid. This includes metadata extraction and consolidation against CrossRef and PubMed, structuring the text body and bibliographical references.

1.2. if the format is an publisher XML format (see Pub2TEI for the list of supported XML formats, e.g. TEI, JATS/NLM, ScholarOne, BMJ, Elsevier staging format, OUP, PNAS, RSC, Sage, Wiley, etc.), Pub2TEI converts the XML to the same customised structured TEI representation as GROBID.

  1. The document body is then segmented into sentences thanks to the Pragmatic Segmenter or OpenNLP, with some customization to better support scientific texts (i.e. avoiding wrong sentence break in the middle of reference callout or in the middle of scientific notations, and taking into account section and paragraph breaks as identified in the structure recognition step in GROBID).

  2. Each sentence is going through a cascade of text classifiers, typically all based on a fine-tuned SciBERT deep learning architecture integrated in Java via DeLFT and JEP, to predict if the sentence introduce a dataset, and if yes, which dataset type and sub type is introduced.

  3. The text body is then processed by a sequence labeling model which aims at recognizing the section relevant to dataset introductions and presentations (zoning of "data" sections). "Materials and Methods" for instance is a usual relevant section, but other sections might be relevant and the "Materials and Methods" sections can appeared with a variety of section headers and subsections not relevant. This sequence labelling process is realized currently by a CRF using various features including the predictions produced in the previous steps 3.

  4. A final selection of the predicted datasets takes place for the sections identified as introducing potentially datasets, using the result of the sentence classification of step 3 for predicting additionally the type and subtype of the recognized datasets.

The result of the service is a TEI file representing the article, enriched with sentence boundaries and predicted data set information.

Fluorometry

Above, the Fluorometry dataset class word cloud.

Training

The DataSeer dataset covers:

After alignment with the actual content of the full article bodies (via Grobid) for the first set, the data is used for training the recognition of sections introducing datasets (so called "zoning" task implemented with CRF using the Wapiti library), a binary classifier (sentence mentioning a dataset or not) and the data type and subtype classifiers (SciBERT).

The total amount of annotated sentences presenting dataset is 25,603. The rest of the sentences of the 3,000 annotated articles can be used as negative examples via sampling techniques.

An additional "data reuse" model can also be trained to predict if an identified dataset is newly introduced or reused in the context of the article, using around 400 positive "reuse" examples (on the life science domain, our annotated data indicates only 3.6% of reused datasets amongs all dataset mentioned).

Docker Image

A docker image for the dataseer-ml service can be used/built with the project Dockerfile. This is the simplest and preferred way to run the dataseer-ml service. The GPUs on your system will be automatically recognized an used, with fallback to CPU if no GPU available. Note that the system works okay with CPU only (10-30 seconds per article), but the runtime is obviously considerably better with GPU.

For offline processing (e.g. non interactive usage scenario), it is recommended to exploit parallelization as much as possible, because the service takes advantage of multi-threading with CPU-only or GPU. Concurrent processing of PDF, XML or text input should be adjusted to the server capabilities.

Use the Docker image available on Docker Hub

dataseer-ml service is available as a Docker image on docker HUB, pull the image (here the lastest available version, change the version number as needed) as follow:

docker pull grobid/dataseer:0.7.3

After pulling or building the Docker image, you can now run the dataseer-ml service as a container as follow:

docker run --rm --gpus all -it -p 8060:8060 --init grobid/dataseer:0.7.3

Javascript demo/console web app is then accessible at http://localhost:8060. You can change the port mapping for the service at launch - for instance port :8080 as follow:

docker run --rm --gpus all -it -p 8080:8060 --init grobid/dataseer:0.7.3

Build and use the image yourself

The complete process is as follow:

~/grobid/dataseer-ml$ cp ./Dockerfile.dataseer ..
docker build -t grobid/dataseer:0.7.3 --build-arg GROBID_VERSION=0.7.2-SNAPSHOT --file Dockerfile.dataseer .

The Docker image build take several minutes, installing GROBID, dataseer-ml, a complete Python Deep Learning environment based on DeLFT and pre-trained embeddings downloaded from the internet and pre-compiled. The resulting image is very large, more than 16GB, in particular due to the contained embeddings, models and kilometers of Python libraries.

docker run --rm --gpus all -it -p 8060:8060 --init grobid/dataseer:0.7.3

The build image includes the automatic support of GPU when available on the host machine via the parameter --gpus all (with automatic recognition of the CUDA version), with fall back to CPU if GPU are not available. The support of GPU is only available on Linux host machine.

The dataseer-ml service is available at the default host/port localhost:8060, but it is possible to map the port at launch time of the container, e.g. for port :8080:

docker run --rm --gpus all -it -p 8080:8060 --init grobid/dataseer:0.7.3

Build

For building locally dataseer-ml, first install GROBID:

git clone https://github.com/kermitt2/grobid

Install then dataseer-ml and move it as a sub-module of GROBID:

git clone https://github.com/dataseer/dataseer-ml
mv dataseer-ml grobid/

Install DeLFT:

git clone https://github.com/kermitt2/delft

Follow the installation described in the DeLFT documentation. If necessary, update the path to the DeLFT installation in the grobid.yaml file located under grobid-home/config/grobid.properties.

By default, the project can process scientific articles in PDF and TEI formats. To process JATS/NLM, scholarOne and a variety of other native publisher formats, Pub2TEI needs to be installed:

git clone https://github.com/kermitt2/Pub2TEI

If required, update the path to the Pub2TEI installation in the dataseer-ml.yaml file located under resources/config/:

# path to Pub2TEI repository as available at https://github.com/kermitt2/Pub2TEI
pub2teiPath: "../../Pub2TEI/"

Finally, copy the models under grobid-home and build dataseer-ml:

cd grobid/dataseer-ml
./gradlew copyModels
./gradlew clean install

Web service API

Start the service

./gradlew appRun

Service console

Javascript demo/console web app is then accessible at http://localhost:8060. From the console and the RESTfull services tab, you can process chunk of text (select Process text Sentence), process a complete PDF document (select Process PDF), process a TEI document (select Process TEI) or process an XML publisher native document (such as JATS - select Process JATS/NLM/...) .

DataSeer-ml Demo

Process a PDF document

Upload a PDF document, extract its content and convert it into structured TEI (via GROBID), identify dataset introductory section, segment into sentences, identify sentence introducing a dataset and classify the dataset type. Return a TEI representation of the PDF, enriched with Dataseer information.

Example:

curl --form input=@./resources/samples/journal.pone.0198050.pdf localhost:8060/service/processDataseerPDF

Process a TEI document

Upload a TEI document, identify dataset introductory section, segment into sentences, identify sentence introducing a dataset and classify the dataset type. Return the TEI document enriched with Dataseer information. It is assumed that the input TEI document follows the Grobid customization (see here).

Example:

curl --form input=@./resources/samples/journal.pone.0198050.tei.xml localhost:8060/service/processDataseerTEI

Process native publisher XML document

Upload a publisher native XML format document, convert it into structured TEI (via Pub2TEI), identify dataset introductory section, segment into sentences, identify sentence introducing a dataset and classify the dataset type. Return a TEI representation of the PDF, enriched with Dataseer information.

Example:

curl --form input=@./resources/samples/journal.pone.0198050.xml localhost:8060/service/processDataseerJATS

See Pub2TEI for the exact list of supported formats.

Process a sentence in isolation

Identify if the sentence introduces a dataset, if yes classify the dataset type. This service is offered for test and demonstration purposes. Use the document-level service for processing an article for a complete and realistic usage.

Example:

curl -X POST -d "text=This is a sentence." http://localhost:8060/service/processDataseerSentence
curl -GET --data-urlencode "text=This is a another sentence." http://localhost:8060/service/processDataseerSentence

Getting the json datatype resource file

The DataSeer client can access the json file specifying the datatypes supported by the classifers, together with metadata for each data type (description, best data sharing policy, link to the corresponding DataSeer Wiki page, etc.) with the following endpoint:

curl -GET localhost:8060/service/jsonDataTypes

Getting the json datatype resource file after re-sync with the DataSeer Wiki

This service triggers a web crawling of the DataSeer Wiki pages describing the supported data types. Metadata about each type are extracted (description, best data sharing policy, link to the corresponding DataSeer Wiki page, etc.) and a json datatype resource file is assembled and served to the client:

curl -GET localhost:8060/service/resyncJsonDataTypes

Training data assembling and generating classification models

Importing and assembling training data created from scratch

Form this source, training data is available in a tabular format with reference to Open Access articles. The following process will align these tabular data (introduced by parameter -Pcsv) with the actual article content (JATS/NLM and PDF via GROBID) to create a full training corpus.

./gradlew annotated_corpus_generator_csv -Ppdf=PATH/TO/THE/FULL/TEXTS/PDF -Pfull=PATH/TO/THE/FULL/TEXTS/NLM/ -Pcsv=PATH/TO/THE/TABULAR/TRAINING/DATA -Pxml=PATH/WHERE/TO/WRITE/THE/ASSEMBLED/TRAINING/DATA

For instance:

./gradlew annotated_corpus_generator_csv -Ppdf=/mnt/data/resources/plos/pdf/ -Pfull=/mnt/data/resources/plos/nlm/ -Pcsv=/home/lopez/grobid/dataseer-ml/resources/dataset/dataseer/csv/ -Pxml=/home/lopez/grobid/dataseer-ml/resources/dataset/dataseer/corpus/

Some reports will be generated to describe the alignment failures.

Training the classification models

The classifier models are relying on the DeLFT deep learning library, which is integrated in Grobid.

After assembling the training data, the classification models can be trained with the following command under the DeFLT project (curren version 0.3.1 of DeLFT):

cd delft
python3 delft/applications/dataseerClassifier.py train --architecture bert --transformer allenai/scibert_scivocab_cased

Possible architectures are documented in the DeLFT project.

For producing an evaluation (including n-fold cross evaluation), see the DeLFT documentation.

[To Be Completed]

Training the dataset-relevant section identifier model

This model is a sequence labeling model working at segment-level (e.g. sequence of segments and one label per segment).

Train with all available training data (default grobid-home path is grobid/grobid-home so usually no need to indicate this parameter):

./gradlew train_dataseer -PgH=/path/grobid/home

Evaluation with a random split of the annotated data with a ratio of 0.9 (90% training, 10% evaluation):

./gradlew eval_dataseer_split -PgH=/path/grobid/home -Ps=0.9

10-fold cross-evaluation:

./gradlew eval_dataseer_nfold -PgH=/path/grobid/home -Pt=10

Training data from the DataSeer web application

The dataset annotations performed with the DataSeer web application are stored directly in a TEI format. We have thus in the TEI document at the same time manually corrected dataset annotations and the exact contexts of mention of the dataset in the structured document. We can therefore add this data to the existing training data and retrain the models - the DataSeer web application being actually also a PDF-annotation tool for new creating training data.

To generate training data from the application, first indicate the connection information to access the DataSeer web API (file config.json). A token corresponding to the curator level user right is necessary (it can be generated from the DataSeer web application, in the account panel). Then indicate in the config file the usage names corresponding to annotators/curators that you wish to consider to retrieve annotated valid documents. The latest versions of the datasets of the documents modified by the list of indicated annotators/curators will be used as extracted training data.

By default the script outputs the data "valided" by the indicated annotators or curators (who is providing an expert validation on the manual annotations). If relevant, you can modify the script to apply other criteria of selection. Then use the script as follow:

> python3 app_document_converter.py --config my_config.json --output ~/tmp/ 

The command will produce 3 files in the cvs training data format:

In addition, a CVS file containing all the previous fields and some complementary ones called extract_summary.csv will also generated, not for the purposes of training, but for human review (it includes additional information not used for training the machine learning models, such as dataset names, dataset permanent identifier, etc.).

Finally, the corresponding TEI files will be exported and written in a subdirector corpus/ under the directory specified by the --output parameter. These TEI files can then be used as such to retrain the dataset-relevant section identifier model (see previous section, these new TEI files needs to be copied under dataseer-ml/resources/dataset/dataseer/corpus/).

This process enables in practice a continuous re-training of the 4 different ML models based on the decisions/corrections of the end-users of the application.

Benchmarking

Here are some benchmarkings on the dataset recognition and data type classification tasks. Given the current sparsity of the training examples for some data types, only a subset of major data types can be predicted as of mid-2020.

The evaluated classification models are:

SciBERT provides almost always the best classification accuracy.

Reported scores are obtained with 10-fold cross-validation.

Tasks and evaluations:

Trained initially with 21,042 examples (approx. 55% positive, 45% negative).

Initial model comparison:

BiGRU
-----
                   precision        recall       f-score       support
       dataset        0.8619        0.9465        0.9022          1121
    no_dataset        0.9198        0.8019        0.8568           858

bert-base-en
------------
                   precision        recall       f-score       support
       dataset        0.8466        0.9795        0.9082          1121
    no_dataset        0.9663        0.7681        0.8558           858 

SciBERT
-------
                   precision        recall       f-score       support
       dataset        0.9053        0.9233        0.9142          1108
    no_dataset        0.8975        0.8743        0.8857           851

Balancing more realistically positive and negative in the training and evaluation set (approx. 30% positive, 70% negative):

SciBERT 
-------
Evaluation on 3574 instances:
                   precision        recall       f-score       support
       dataset        0.8844        0.9428        0.9127          1136
    no_dataset        0.9725        0.9426        0.9573          2438

Results (10-2020) after extending the training data to around 59,400 examples (approx. 30% positive, 70% negative):

SciBERT 
-------
Evaluation on 5993 instances:
                   precision        recall       f-score       support
       dataset        0.9166        0.9664        0.9408          2320
    no_dataset        0.9780        0.9445        0.9609          3673

Results (04-2022) using DeLFT 0.3.1 updated architecture based on TensorFlow 2.7, around 59,400 examples (approx. 30% positive, 70% negative):

SciBERT 
-------
Evaluation on 5993 instances:
                   precision        recall       f-score       support
       dataset        0.9339        0.9560        0.9448          2320
    no_dataset        0.9718        0.9573        0.9645          3673

Results (08-2023) using DeLFT 0.3.3, around 112,200 examples (approx. 30% positive, 70% negative):

SciBERT 
-------
Evaluation on 11307 instances:
                   precision        recall       f-score       support
       dataset        0.9173        0.9428        0.9299          3881
    no_dataset        0.9697        0.9556        0.9626          7426
BiGRU
-----
                   precision        recall       f-score       support
   Angiography        0.0000        0.0000        0.0000             1
   Calorimetry        0.0000        0.0000        0.0000             1
Chromatography        0.6667        0.5455        0.6000            11
Coulombimetry         0.0000        0.0000        0.0000             0
       Dataset        0.4828        0.6222        0.5437            45
  Densitometry        0.0000        0.0000        0.0000             0
Digital Drople        0.0000        0.0000        0.0000             0
Electrocardiog        0.0000        0.0000        0.0000             3
Electroencepha        1.0000        1.0000        1.0000             2
Electromyograp        0.0000        0.0000        0.0000             3
Electrooculogr        0.0000        0.0000        0.0000             1
Electrophysiol        0.0000        0.0000        0.0000             0
Electroretinog        0.0000        0.0000        0.0000             0
Emission flame        0.0000        0.0000        0.0000             1
Flow cytometry        0.9444        0.8095        0.8718            21
  Genetic Data        0.7879        0.6341        0.7027            41
         Image        0.7875        0.8289        0.8077           152
Mass Spectrome        0.0000        0.0000        0.0000             4
  Protein Data        0.0000        0.0000        0.0000             1
Real-Time Poly        0.8286        0.8788        0.8529            33
    Sound data        0.0000        0.0000        0.0000             1
  Spectrometry        0.7308        0.7917        0.7600            48
Spectrum Analy        0.0000        0.0000        0.0000             0
Spirometry dat        0.0000        0.0000        0.0000             0
  Tabular data        0.8156        0.8048        0.8102           753
Video Recordin        0.0000        0.0000        0.0000             2
Voltammetry da        0.0000        0.0000        0.0000             1
X-Ray Diffract        0.0000        0.0000        0.0000             7
X-Ray fluoresc        0.0000        0.0000        0.0000             0
    no_dataset        0.8459        0.8663        0.8560           830

bert-base-en
------------
                   precision        recall       f-score       support
   Angiography        0.0000        0.0000        0.0000             1
   Calorimetry        0.0000        0.0000        0.0000             1
Chromatography        0.0000        0.0000        0.0000            11
Coulombimetry         0.0000        0.0000        0.0000             0
       Dataset        0.7368        0.3111        0.4375            45
  Densitometry        0.0000        0.0000        0.0000             0
Digital Drople        0.0000        0.0000        0.0000             0
Electrocardiog        0.0000        0.0000        0.0000             3
Electroencepha        0.0000        0.0000        0.0000             2
Electromyograp        0.0000        0.0000        0.0000             3
Electrooculogr        0.0000        0.0000        0.0000             1
Electrophysiol        0.0000        0.0000        0.0000             0
Electroretinog        0.0000        0.0000        0.0000             0
Emission flame        0.0000        0.0000        0.0000             1
Flow cytometry        0.9091        0.4762        0.6250            21
  Genetic Data        0.5490        0.6829        0.6087            41
         Image        0.7204        0.8816        0.7929           152
Mass Spectrome        0.0000        0.0000        0.0000             4
  Protein Data        0.0000        0.0000        0.0000             1
Real-Time Poly        0.6667        0.9697        0.7901            33
    Sound data        0.0000        0.0000        0.0000             1
  Spectrometry        0.7049        0.8958        0.7890            48
Spectrum Analy        0.0000        0.0000        0.0000             0
Spirometry dat        0.0000        0.0000        0.0000             0
  Tabular data        0.7670        0.8964        0.8267           753
Video Recordin        0.0000        0.0000        0.0000             2
Voltammetry da        0.0000        0.0000        0.0000             1
X-Ray Diffract        0.0000        0.0000        0.0000             7
X-Ray fluoresc        0.0000        0.0000        0.0000             0
    no_dataset        0.9391        0.7988        0.8633           830

SciBERT
-------
                   precision        recall       f-score       support
   Calorimetry        0.0000        0.0000        0.0000             2
Chromatography        0.6000        1.0000        0.7500             6
Coulombimetry         0.0000        0.0000        0.0000             0
  Densitometry        0.0000        0.0000        0.0000             0
Electrocardiog        0.5000        0.5000        0.5000             4
Electroencepha        0.0000        0.0000        0.0000             1
Electromyograp        0.0000        0.0000        0.0000             1
Electrooculogr        0.0000        0.0000        0.0000             0
Electrophysiol        0.0000        0.0000        0.0000             0
Electroretinog        0.0000        0.0000        0.0000             1
Emission Flame        0.0000        0.0000        0.0000             0
Flow Cytometry        0.9375        0.9375        0.9375            16
  Genetic Data        0.6535        0.8354        0.7333            79
         Image        0.7433        0.8968        0.8129           155
Mass Spectrome        0.9048        0.9048        0.9048            21
  Protein Data        0.0000        0.0000        0.0000             3
    Sound Data        0.0000        0.0000        0.0000             2
  Spectrometry        0.7021        0.8919        0.7857            37
Spectrum Analy        0.0000        0.0000        0.0000             0
Systematic Rev        0.0000        0.0000        0.0000             2
  Tabular Data        0.8524        0.8620        0.8571           797
Video Recordin        0.0000        0.0000        0.0000             5
Voltammetry Da        0.0000        0.0000        0.0000             0
X-Ray Diffract        0.0000        0.0000        0.0000             4
    no_dataset        0.9685        0.9463        0.9573          2438

Extending the training data to 3000 articles (10-2020):

SciBERT
-------

Total 47669 instances

                   precision        recall       f-score       support
   calorimetry        0.0000        0.0000        0.0000             1
chromatography        0.7532        0.8056        0.7785            72
coulombimetry         0.0000        0.0000        0.0000             0
  densitometry        0.0000        0.0000        0.0000             0
electrocardiog        0.0000        0.0000        0.0000             5
electroencepha        0.8000        0.8000        0.8000             5
electromyograp        0.0000        0.0000        0.0000             0
electrooculogr        0.0000        0.0000        0.0000             0
electrophysiol        0.0000        0.0000        0.0000             1
electroretinog        0.0000        0.0000        0.0000             1
emission flame        0.0000        0.0000        0.0000             0
flow cytometry        0.8971        0.8841        0.8905            69
  genetic data        0.8259        0.9022        0.8623           184
         image        0.8041        0.9105        0.8540           257
mass spectrome        0.6667        0.6562        0.6614            64
  protein data        0.0000        0.0000        0.0000             0
    sound data        1.0000        0.2500        0.4000             4
  spectrometry        0.7544        0.8866        0.8152            97
spectrum analy        0.0000        0.0000        0.0000             0
systematic rev        0.0000        0.0000        0.0000             1
  tabular data        0.8772        0.9087        0.8927          1588
video recordin        0.0000        0.0000        0.0000             3
voltammetry da        0.0000        0.0000        0.0000             1
x-ray diffract        0.0000        0.0000        0.0000             8

Results 08-2023,

SciBERT
-------

Total 63,912 instances

Evaluation on 6113 instances:
                   precision        recall       f-score       support
   calorimetry        0.6667        0.6667        0.6667             3
chromatography        0.8731        0.9141        0.8931           128
 code software        0.6629        0.8227        0.7342           141
coulombimetry         0.0000        0.0000        0.0000             0
dataset re-use        0.0000        0.0000        0.0000             2
  densitometry        0.0000        0.0000        0.0000             1
electrocardiog        0.7500        1.0000        0.8571             3
electroencepha        0.5000        1.0000        0.6667             2
electromyograp        0.5000        1.0000        0.6667             1
electrooculogr        0.0000        0.0000        0.0000             0
electrophysiol        0.0000        0.0000        0.0000             2
electroretinog        0.0000        0.0000        0.0000             0
emission flame        0.0000        0.0000        0.0000             0
flow cytometry        0.8471        0.7273        0.7826            99
  genetic data        0.8529        0.8315        0.8421           279
         image        0.7988        0.8133        0.8060           332
 lab materials        0.7949        0.8757        0.8333           177
mass spectrome        0.6863        0.7071        0.6965            99
    no_dataset        0.9519        0.9241        0.9378          2357
         other        0.0000        0.0000        0.0000             0
  protein data        0.5882        0.8333        0.6897            12
      protocol        0.5294        0.3000        0.3830            30
    sound data        0.8000        0.6667        0.7273             6
  spectrometry        0.8462        0.8209        0.8333           134
spectrum analy        0.0000        0.0000        0.0000             2
systematic rev        0.0000        0.0000        0.0000             5
  tabular data        0.8911        0.9063        0.8986          2274
video recordin        0.5714        0.4444        0.5000             9
voltammetry da        0.0000        0.0000        0.0000             1
x-ray diffract        0.7368        1.0000        0.8485            14
BiGRU
-----

Evaluation Chromatography subtypes
Evaluation on 12 instances:
                   precision        recall       f-score       support
High Pressure Liq. 0.5833           1.0000        0.7368             7


Evaluation Genetic Data subtypes
Evaluation on 46 instances:
                   precision        recall       f-score       support
High-Throughpu        0.7500        0.9000        0.8182            10
Sequence Analy        0.4118        1.0000        0.5833            14


Evaluation Image subtypes
Evaluation on 145 instances:
                   precision        recall       f-score       support
Electrophoresi        0.7500        0.9600        0.8421            25
    Microscopy        0.9577        0.9189        0.9379            74
Magnetic Reson        0.3158        0.7500        0.4444             8
           nan        0.5714        0.6667        0.6154            18

Evaluation Mass Spectrometry subtypes
Evaluation on 5 instances:
                   precision        recall       f-score       support
Gas Chromatogr        0.6000        1.0000        0.7500             3

Evaluation Spectrometry subtypes
Evaluation on 51 instances:
                   precision        recall       f-score       support
Spectrophotome        0.5490        1.0000        0.7089            280

Evaluation Tabular data subtypes
Evaluation on 762 instances:
                   precision        recall       f-score       support
           nan        0.8148        0.8894        0.8505           470
         Assay        0.7288        0.6719        0.6992            64
   Fluorometry        0.9000        0.6000        0.7200            15
  Sample Table        0.3889        0.1944        0.2593            36
Subject Data T        0.7687        0.6975        0.7314           162


bert-base-en
------------

Evaluation Chromatography subtypes
                   precision        recall       f-score       support
High Pressure Liq. 0.5455        1.0000        0.7059             6


Evaluation Dataset subtypes
                   precision        recall       f-score       support
Existing datas        0.9070        1.0000        0.9512            39


Evaluation Genetic Data subtypes
                   precision        recall       f-score       support
High-Throughpu        0.4091        1.0000        0.5806             9
Sequence Analy        0.4375        0.4375        0.4375            16


Evaluation Image subtypes
                   precision        recall       f-score       support
Computerized T        0.3500        0.7778        0.4828             9
Electrophoresi        0.9062        0.9667        0.9355            30
    Microscopy        0.8158        1.0000        0.8986            62
Magnetic Reson        1.0000        0.0833        0.1538            12
           nan        0.6667        0.2353        0.3478            17


Evaluation Mass Spectrometry subtypes
                   precision        recall       f-score       support
Gas Chromatogr        0.5000        0.6667        0.5714             3


Evaluation Spectrometry subtypes
                   precision        recall       f-score       support
Spectrophotome        0.6667        1.0000        0.8000            22


Evaluation Tabular data subtypes
                   precision        recall       f-score       support
           nan        0.8065        0.8789        0.8412           479
         Assay        0.6711        0.6711        0.6711            76
   Fluorometry        0.6129        0.8636        0.7170            22
  Sample Table        0.5333        0.2222        0.3137            36
Subject Data T        0.7628        0.7212        0.7414           165


SciBERT
-------

Evaluation Chromatography subtypes
                   precision        recall       f-score       support
High Pressure         0.6364        1.0000        0.7778             7


Evaluation Genetic Data subtypes
                   precision        recall       f-score       support
Real-Time Poly        0.8333        1.0000        0.9091            30
High-Throughpu        0.9286        0.8667        0.8966            15
Sequence Analy        0.8235        0.7778        0.8000            18


Evaluation Image subtypes
                   precision        recall       f-score       support
X-Ray Computed        0.8182        0.6923        0.7500            13
Electrophoresi        0.9286        1.0000        0.9630            26
    Microscopy        0.9200        0.9718        0.9452            71


Evaluation Mass Spectrometry subtypes
                   precision        recall       f-score       support
Liquid Chromat        0.5455        1.0000        0.7059             6


Evaluation Spectrometry subtypes
                   precision        recall       f-score       support
Spectrophotome        0.5385        1.0000        0.7000            21


Evaluation Tabular data subtypes
                   precision        recall       f-score       support
           nan        0.8963        0.7840        0.8364           551
         Assay        0.6477        0.7808        0.7081            73
   Fluorometry        0.8125        0.8667        0.8387            15
  Sample Table        0.4211        0.4444        0.4324            36
Subject Data T        0.6054        0.8766        0.7162           154

Results 04-2022 using DeLFT 0.3.1 updated architecture based on TensorFlow 2.7, 11,500 annotated sentences:

Evaluation on 1122 instances:
                   precision        recall       f-score       support
      no_reuse        0.9907        0.9871        0.9889          1083
         reuse        0.6744        0.7436        0.7073            39

Results 08-2023 using DeLFT 0.3.3, 27,100 annotated sentences:

Evaluation on 2713 instances:
                   precision        recall       f-score       support
      no_reuse        0.9307        0.9473        0.9389          2183
         reuse        0.7658        0.7094        0.7365           530

Additional convenient scripts

Load training data documents in the dataseer web application

This is only relevant to examine the manually labeled training data, and optionally correct it via the existing web application (cool stuff: documents with datasets inputted via the web application has the same format as the training data, thus user-annotated documents via the web application dataseer-web can be used for training by datasser-ml). All the documents present in the local training data repository (after importing the training, see above) under dataseer-ml/resources/dataset/dataseer/corpus/ will be loaded via the dataseer web API.

cd scripts/

node loader.js

Convert data type information from the DataSeer Doku Wiki to JSON

The following Python script converts the data type specification from the DataSeer Doku Wiki (http://wiki.dataseer.io) into a JSON representation used by the DataSeer Web application. In addition, it will use the training file(s) to inject counts for each datatype. These frequency information can are used by the DataSeer Web application to provide a default ranking of datatypes in the drop down menus when a datatype is assigned manually.

cd scripts/

pip3 install pandas beautifulsoup4

python3 converter.py ../resources/DataTypes.csv ../resources/DataTypes.json

Note that at the end of the DataSeer Doku Wiki conversion, the script will report data types in the training data inconsistent with the DataSeer Doku Wiki. Those data types must be reviewed and updated to be consistent with Wiki, and the machine learning models must be retrained with the updated training data to produce the new data types.

Contact and License

Author and contact: Patrice Lopez (patrice.lopez@science-miner.com)

The development of dataseer-ml was supported by a Sloan Foundation grant, see here

dataseer-ml is distributed under Apache2 license.