Home

Awesome

OSIRRC Docker Image for Indri

Generic badge DOI

Claudia Hauff

This is the docker image for Indri v5.13 conforming to the OSIRRC jig for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019. This image is available on Docker Hub. The OSIRRC 2019 image library contains a log of successful executions of this image.

Quick Start

First, clone the jig and follow its setup instructions - in particular, do not forget to also clone and install trec_eval inside the jig directory!

Indexing a collection

Once that is done, the following jig command can be used to index TREC disks 4/5 for robust04:

python3 run.py prepare \
  --repo osirrc2019/indri \
  --tag v0.1.0 \
  --collections robust04=/path/to/disk45=trectext

The core18 indexing command is very similar:

python3 run.py prepare \
  --repo osirrc2019/indri \
  --collections core18=/path/to/WashingtonPost.v2=trectext

For gov2, we have to change the corpus format to trecweb (note: format trectext will not throw an error but instead lead to a poorly performing index as HTML documents are not handled correctly):

python3 run.py prepare \
  --repo osirrc2019/indri \
  --collections gov2=/path/to/gov2=trecweb

Searching a collection

The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection:

python3 run.py search \
  --repo osirrc2019/indri \
  --tag v0.1.0 \
  --output out/indri \
  --qrels qrels/qrels.robust04.txt \
  --topic topics/topics.robust04.txt \
  --collection robust04 \ 
  --top_k 1000 \
  --opts out_file_name="robust.dir1000" rule="method:dirichlet,mu:1000" topic_type="title"

For core18 use:

python3 run.py search \
  --repo osirrc2019/indri \
  --output out/indri \
  --qrels qrels/qrels.core18.txt \
  --topic topics/topics.core18.txt \
  --collection core18 \ 
  --top_k 1000 \
  --opts out_file_name="core18.dir1000" rule="method:dirichlet,mu:1000" topic_type="title"

For gov2 use:

python3 run.py search \
  --repo osirrc2019/indri \
  --output out/indri \
  --qrels qrels/qrels.701-850.txt \
  --topic topics/topics.701-850.txt \
  --collection gov2 \ 
  --top_k 1000 \
  --opts out_file_name="gov2.dir1000" rule="method:dirichlet,mu:1000" topic_type="title"

The option --opts has a rule entry to determine the retrieval method, a topic_type entry to determine the TREC topic type (either title, desc or narr or a combination, e.g. title+desc), a use_prf entry to determine the usage of pseudo-relevance feedback ("1" to use PRF, anything else to not use it) and a sd entry to switch on ("1") or off the sequence dependency model (sd will have no effect on tfidf or okapi). Both sd and use_prf are optional parameters; if they are not provided, they are by default switched off.

Valid retrieval rule entries can be found in the Indri documentation. Importantly, the tfidf and okapi methods also work in the rule entry (i.e. no separate baseline entry is necessary). Note that there are default parameter settings for all of the retrieval rules - if only the name, but no further information is provided, the defaults are used.

While a retrieval rule can be specified, for PRF this option is not available at the moment (check searchRobust04.sh to make changes), instead a set of default settings are used: 50 feedback documents, 25 feedback terms and equal weighting between the original and expanded query.

Setup Details

Expected Results

The following table contains examples of --opts and the expected retrieval effectiveness in MAP, Precision@30/@10 and NDCG@20. These metrics may seem arbitrary; in fact, they were chosen to make---at least on paper---comparisons to published works on the same corpus.

robust04

The results below are computed based on the 250 topics of robust04.

MAPP@30P@10NDCG@20
:one: --opts out_file_name="robust.dir1000.title" rule="method:dirichlet,mu:1000" topic_type="title"0.24990.31000.42530.4201
:two: --opts out_file_name="robust.dir1000.title.sd" rule="method:dirichlet,mu:1000" topic_type="title" sd="1"0.25470.31460.43290.4232
:three: --opts out_file_name="robust.dir1000.title.prf" rule="method:dirichlet,mu:1000" topic_type="title" use_prf="1"0.28120.32480.43860.4276
:four: --opts out_file_name="robust.dir1000.title.prf.sd" rule="method:dirichlet,mu:1000" topic_type="title" use_prf="1" sd="1"0.28550.32950.44740.4298
:five: --opts out_file_name="robust.jm0.5.title" rule="method:linear,collectionLambda:0.5" topic_type="title"0.22420.28390.38190.3689
:six: --opts out_file_name="robust.bm25.title" rule="okapi,k1:1.2,b:0.75" topic_type="title"0.23380.29950.41810.4041
:seven: --opts out_file_name="robust.bm25.title.prf" rule="okapi,k1:1.2,b:0.75" topic_type="title" use_prf="1"0.25630.30410.40120.3995
:eight: --opts out_file_name="robust.bm25.title+desc" rule="okapi,k1:1.2,b:0.75" topic_type="title+desc"0.27020.32740.46180.4517
:nine: --opts out_file_name="robust.dir1000.title+desc.prf.sd" rule="method:dirichlet,mu:1000" topic_type="title+desc" use_prf="1" sd="1"0.29710.35620.45500.4448
:keycap_ten: --opts out_file_name="robust.dir1000.desc" rule="method:dirichlet,mu:1000" topic_type="desc"0.20230.25810.37030.3635

How do we fare compared to Indri results reported in papers? Here, we are considering the Robust04 numbers reported by researchers working at the institute developing/maintaining Indri:

Conclusion: the sequential dependency model outperforms the standard language modeling approach, though the absolute differences we observe are smaller. BM25 peforms worse than expected, this is likely a hyperparameter issue (we did not tune, used default values). The biggest difference can be found in the results involving TREC topic descriptions, a preprocessing issue (specific stopword phrases for description topics).

gov2

The results below are computed based on the 150 topics (topics/topics.701-850.txt) available for gov2.

MAPP@30P@10NDCG@20
:one: --opts out_file_name="gov2.dir1000.title" rule="method:dirichlet,mu:1000" topic_type="title"0.28000.47410.51740.4353
:two: --opts out_file_name="gov2.dir1000.title.sd" rule="method:dirichlet,mu:1000" topic_type="title" sd="1"0.29040.48990.54630.4507
:three: --opts out_file_name="gov2.dir1000.title.prf" rule="method:dirichlet,mu:1000" topic_type="title" use_prf="1"0.30330.48330.52480.4276
:four: --opts out_file_name="gov2.dir1000.title.prf.sd" rule="method:dirichlet,mu:1000" topic_type="title" use_prf="1" sd="1"0.31040.48630.52280.4280
:five: --opts out_file_name="gov2.jm0.5.title" rule="method:linear,collectionLambda:0.5" topic_type="title"0.18080.30860.31610.2604
:six: --opts out_file_name="gov2.bm25.title" rule="okapi,k1:1.2,b:0.75" topic_type="title"0.24850.44860.49730.4099
:seven: --opts out_file_name="gov2.bm25.title.prf" rule="okapi,k1:1.2,b:0.75" topic_type="title" use_prf="1"0.25650.45540.50000.3987
:eight: --opts out_file_name="gov2.bm25.title+desc" rule="okapi,k1:1.2,b:0.75" topic_type="title+desc"0.27050.47490.52750.4441
:nine: --opts out_file_name="gov2.bm25.title+desc.prf.sd" rule="method:dirichlet,mu:1000" topic_type="title+desc" use_prf="1" sd="1"0.26210.44830.48520.4040
:keycap_ten: --opts out_file_name="gov2.dir1000.desc" rule="method:dirichlet,mu:1000" topic_type="desc"0.13360.32440.36580.3096

core18

The results below are computed based on the 50 topics (topics/topics.core18.txt) available for core18.

MAPP@30P@10NDCG@20
:one: --opts out_file_name="core18.dir1000.title" rule="method:dirichlet,mu:1000" topic_type="title"0.23320.32600.44200.3825
:two: --opts out_file_name="core18.dir1000.title.sd" rule="method:dirichlet,mu:1000" topic_type="title" sd="1"0.24280.33600.46600.3970
:three: --opts out_file_name="core18.dir1000.title.prf" rule="method:dirichlet,mu:1000" topic_type="title" use_prf="1"0.28000.37930.47200.4219
:four: --opts out_file_name="core18.dir1000.title.prf.sd" rule="method:dirichlet,mu:1000" topic_type="title" use_prf="1" sd="1"0.28160.38870.47200.4190
:five: --opts out_file_name="core18.jm0.5.title" rule="method:linear,collectionLambda:0.5" topic_type="title"0.20550.29200.37800.3349
:six: --opts out_file_name="core18.bm25.title" rule="okapi,k1:1.2,b:0.75" topic_type="title"0.22810.32800.41800.3718
:seven: --opts out_file_name="core18.bm25.title.prf" rule="okapi,k1:1.2,b:0.75" topic_type="title" use_prf="1"0.26990.37930.45800.3920
:eight: --opts out_file_name="core18.bm25.title+desc" rule="okapi,k1:1.2,b:0.75" topic_type="title+desc"0.24570.33930.46000.3937
:nine: --opts out_file_name="core18.bm25.title+desc.prf.sd" rule="method:dirichlet,mu:1000" topic_type="title+desc" use_prf="1" sd="1"0.29050.39670.47400.4129
:keycap_ten: --opts out_file_name="core18.dir1000.desc" rule="method:dirichlet,mu:1000" topic_type="desc"0.16740.25870.34000.2957

Implementation

The following is a quick breakdown of what happens in each of the scripts in this repo.

Dockerfile

The Dockerfile installs dependencies (python3, etc.), copies scripts to the root dir, and sets the working dir to /work. It also installs Indri.

index

The robust04 corpus is difficult to handle for Indri: the original version is .z compressed, a compression Indri cannot handle. And thus, indexRobust04.sh first uncompresses the collection, filters out undesired folders (such as cr), downloads Lemur's stopword list, creates a parameter file for indexing and finally calls Indri's IndriBuildIndex.

The core18 corpus is provided in JSON format and needs to be converted to Indri's format in a first step - the nodejs script parseCore18.js does the job.

In contrast, the formatting of gov2 is well suited for Indri, apart from creating a parameter file and calling Indri's `IndriBuildIndex, not much happens.

search

The robust04 and gov2 topics are in "classic" TREC format, a format Indri cannot handle. Thus, first the TREC topic file has to be converted into a format Indri can parse (done in topicFormatting.pl). Note: different retrieval methods require differently formatted Indri topic files. A parameter file is created and finally IndriRunQuery is executed.

The core18 topics are very similar to robust04/gov2 but did require some additional processing (core18 topics contain closing tags that are not present in the robust04 and gov2 topic files).

Reviews