Home

Awesome

OSIRRC Docker Image for Terrier

Build Status Docker Cloud Build Status DOI

Arthur Câmara and Craig Macdonald

This is the docker image for the Terrier toolkit (v5.2) conforming to the OSIRRC jig for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019. This image is available on Docker Hub

Quick Start

The following jig command can be used to index TREC disks 4/5 for robust04:

python run.py prepare --repo osirrc2019/terrier --tag vx.y.z --collections robust04=/tmp/disk45/=trectext

The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection, using BM25 as ranker:

python run.py search  \
    --repo osirrc2019/terrier \
    --tag vx.y.z \
    --collection robust04 \
    --topic topics/topics.robust04.txt \
    --qrels qrels/qrels.robust04.txt\
    --output /tmp/runs

Retrieval Methods:

This image supports the following weighting models: BM25 (bm25), PL2 (pl2) and DPH (dph).

Additionally, it supports Query Expansion and Proximity-based (DFRD) search, by including qe, prox or prox_qe to the --opts config argument: --opts config=<retrieval_model>_<extra>:

(BM25)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=bm25

(BM25 + query expansion)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=bm25_qe

(BM25 + Proximity)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=bm25_prox

(BM25 + Proximity + query expansion)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=bm25_prox_qe

(PL2)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=pl2

(PL2 + query expansion)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=pl2_qe

(PL2 + Proximity)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=pl2_prox

(PL2 + Proximity + query expansion)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=pl2_prox_qe

(DPH)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=dph

(DPH + query expansion)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=dph_qe

(DPH + Proximity)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=dph_prox

(DPH + Proximity + query expansion)

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=dph_prox_qe

NOTE: for running DFRD (Proximity-based model), the index must be build using the --opts=block.indexing=true param

Learning to Rank Runs

Learning-to-rank will typically require that the index has more information, e.g. fields or blocks.

Indexing:

python run.py prepare     --repo osirrc2019/terrier --tag vx.y.z   --collections robust04=/tmp/disk45/=trectext --opts "FieldTags.process=HEADLINE"

Training:

You need to specify the features to be used by Terrier - see http://terrier.org/docs/v5.1/learning.html for more information about Terrier feature definitions.

python run.py train  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt    --test_split $PWD/sample_training_validation_query_ids/robust04_test.txt  --validation_split $PWD/sample_training_validation_query_ids/robust04_validation.txt --model_folder /tmp/runs --opts features="SAMPLE;WMODEL:SingleFieldModel(BM25,0);QI:SingleFieldModel(Dl,0)"

Retrieval:

You will need to specify the bm25_ltr_jforest configuration.

python run.py search  --repo osirrc2019/terrier --tag vx.y.z --collection robust04  --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt   --output /tmp/runs --opts config=bm25_ltr_jforest

Expected Results

robust04

MAPBM25+QE+Prox+Prox + QEDPH+ QE+Prox+Prox +QEPL2+QE
TREC 2004 Robust Track Topics0.23630.27620.24040.27810.24790.28210.25010.28690.22410.2538

core18

MAPBM25+QE+Prox+Prox + QEDPH+ QE+Prox+Prox +QEPL2+QE
TREC 2018 Common Core Track Topics0.23260.29750.23690.29600.24270.30550.24280.30350.22250.2787

GOV2

MAPBM25+QE+Prox+Prox + QEDPH+ QE+Prox+Prox +QEPL2+QE
TREC 2004 Terabyte Track: Topics 701-7500.24610.26210.25370.27150.28040.31200.28340.30640.23340.2478
TREC 2005 Terabyte Track: Topics 751-8000.30810.35060.31260.35070.33110.37540.32550.30950.28840.3160
TREC 2006 Terabyte Track: Topics 801-8500.26290.31180.27240.30850.29170.34940.29040.32880.23630.2739

Interact hooks

This image also supports the interact hooks from the OSIRRC JIG. After initializing the image (with python run.py prepare):

python run.py interact --repo terrier --tag vx.y.z

The following (internal) ports will be made available:

$ /bin/terrier interactive -I http://dockerhost:1981/
terrier query> information retrieval end:5
  Displaying 1-6 results
0 FBIS4-20699 10.268754805435458
1 FBIS4-20702 9.768490153503198
2 FR941027-2-00046 9.491347902606723
3 FBIS4-20701 9.456022500508775
4 FBIS3-24510 9.31403481019499
5 FBIS4-20700 8.79234249484928

NOTE Currently, the JIG may redirect these ports to diverse ones in the host machine. This is due to the way that Docker deals with port assignment. Therefore, you should check the correct port assignment by running:

$> docker ps

and check the right port assignment under the PORTS column.

Reviews