Awesome

LiT5 (List-in-T5) Reranking

RankLLM

We have integrated LiT5 into RankLLM, which is actively maintained and includes additional improvements. We highly recommend using RankLLM.

📟 Instructions

We provide the scripts and data necessary to reproduce reranking results for LiT5-Distill and LiT5-Score on DL19 and DL20 for BM25 and SPLADE++ ED first-stage retrieval. Note you may need to change the batchsize depending on your VRAM. We have observed that results may change slightly when the batchsize is changed. This is a known issue when running inference in bfloat16. Additionally, you may need to remove the --bfloat16 option from the scripts if your GPU does not support it.

Note, the v2 LiT5-Distill models support reranking up to 100 passages at once.

Models

The following is a table of our models hosted on HuggingFace:

Expected Results

This table shows the expected results for reranking with BM25 first-stage retrieval

DL19

DL20

This table shows the expected results for reranking with SPLADE++ ED first-stage retrieval

DL19

DL20

✨ References

If you use LiT5, please cite the following paper: [2312.16098] Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models

@ARTICLE{tamber2023scaling,
  title   = {Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models},
  author  = {Manveer Singh Tamber and Ronak Pradeep and Jimmy Lin},
  year    = {2023},
  journal = {arXiv preprint arXiv: 2312.16098}
}

🙏 Acknowledgments

This repository borrows code from the original FiD repository, the atlas repository, and the RankLLM repository!