Awesome

OSIRRC Docker Image for Anserini

Ryan Clancy and Jimmy Lin

This is the docker image for the Anserini toolkit (v0.5.1) conforming to the OSIRRC jig for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019. This image is available on Docker Hub. The OSIRRC 2019 image library contains a log of successful executions of this image.

• Supported test collections: robust04, core17, core18 (newswire); gov2, cw09b, cw12b (web)
• Supported hooks: init, index, search

Quick Start

The following jig command can be used to index TREC disks 4/5 for robust04:

python run.py prepare \
  --repo osirrc2019/anserini \
  --tag v0.1.1 \
  --collections robust04=/path/to/disk45=trectext

The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection.

python run.py search \
  --repo osirrc2019/anserini \
  --tag v0.1.1 \
  --output out/anserini \
  --qrels qrels/qrels.robust04.txt \
  --topic topics/topics.robust04.txt \
  --collection robust04 \
  --opts search_args="-bm25" out_file_name="run.bm25.robust04"

The arguments search_args and out_file_name can be customized (i.e., search_args="-ql -rm3" and out_file_name="run.ql.rm3.robust04")

Retrieval Methods

The Anserini image supports the following retrieval methods:

• BM25: k1=0.9, b=0.4 (Robertson et al., 1995)
• QL (query likelihood with Dirichlet smooth): mu=1000 (Zhai and Lafferty, 2001)
• +RM3 (RM3 variant of relevance models, applied on top of initial BM25 or QL results): exact parameter settings (Abdul-Jaleel et al., 2004)
• +Ax (Semantic term matching in the axiomatic framework, applied on top of initial BM25 or QL results): exact parameter settings (Fang and Zhai, 2006)

Expected Results

The following numbers should be able to be re-produced using the scripts provided in the bin directory.

robust04

core17

core18

gov2

cw09b

cw12b

Implementation

The following is a quick breakdown of what happens in each of the scripts in this repo.

Dockerfile

The Dockerfile installs dependencies (python3, etc.), copies scripts to the root dir, and sets the working dir to /work.

init

The init script is straightforward - it's simply a shell script (via the #!/usr/bin/env sh she-bang) that invokes wget to download an anserini JAR from Maven Central.

index

The index Python script (via the #!/usr/bin/python3 she-bang) reads a JSON string (see here) containing at least one collection to index (including the name, path, and format). The collection is indexed and placed in a directory, with the same name as the collection, in the working dir (i.e., /work/robust04). At this point, jig takes a snapshot and the indexed collections are persisted for the search hook.

search

The search script reads a JSON string (see here) containing the collection name (to map back to the index directory from the index hook) and topic path, among other options. The retrieval run is performed (using additional --opts params, see above) and output is placed in /output for the jig to evaluate using trec_eval.

References

• Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. (1995) Okapi at TREC-4. TREC
• ChengXiang Zhai and John Lafferty. (2001) A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval. SIGIR.
• Nasreen Abdul-Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Donald Metzler, Mark D. Smucker, Trevor Strohman, Howard Turtle, and Courtney Wade. (2004) UMass at TREC 2004: Novelty and HARD. TREC.
• Hui Fang and ChengXiang Zhai. (2006) Semantic Term Matching in Axiomatic Approaches to Information Retrieval. SIGIR.

Reviews

• Documentation reviewed at commit 68f8291 (06/06/2019) by Chris Kamphuis, Claudia Hauff, Craig Macdonald and Antonio Mallia