Awesome
OSIRRC Docker Image for Birch
Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin
This is the docker image for Birch, a BERT-based experimental IR system, conforming to the OSIRRC jig for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019. The image is available on Docker Hub.
- Supported test collections:
robust04
- Supported hooks:
init
,index
,search
Quick Start
The following jig
command can be used to index TREC disks 4/5 for robust04
:
python run.py prepare \
--repo osirrc2019/birch \
--tag v0.1.0 \
--collections robust04=/path/to/disk45=trectext
The following jig
command can be used to perform a retrieval run as described in Simple Applications of BERT for Document Retrieval:
python run.py search \
--repo osirrc2019/birch \
--tag v0.1.0 \
--collection robust04 \
--topic topics/topics.robust04.txt \
--qrels qrels/qrels.robust04.txt \
--output out/birch \
--measures map P.20 \
--opts experiment=[experiment_name] num_folds=2 anserini_path=anserini tune_params=False
The parameter experiment
can take on the values qa_2cv
, mb_2cv
, qa_5cv
or mb_5cv
, and denotes the pretraining data and cross-validation setting.
Likewise,num_folds
should be set accordingly.
Use tune_params=False
to directly evaluate on the collection, and tune_params=True
to tune the hyperparameters yourself first.
The expected output for experiment=mb_2cv
is as follows:
Evaluating results using trec_eval...
###
# out/birch/run.mb_2cv.cv.a
###
map all 0.3241
P_20 all 0.4217
###
# out/birch/run.mb_2cv.cv.ab
###
map all 0.3240
P_20 all 0.4209
# out/birch/run.mb_2cv.cv.abc
###
map all 0.3244
P_20 all 0.4219
*.a
refer to runs where only the top sentence is considered, *.ab
top 2, and *.abc
top 3 sentences.
Reviews
- Documentation reviewed at commit
4fcfe3c
(2019-06-14) by Ryan Clancy.