Skip to content
Publications

Underline Science Inc. · June 3, 2026

Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation

Association for Computational Linguistics 2026, Abdelrahman Abdallah, Andreas Herzinger, Jamie Holdcroft, Adam Jatowt, Jamshid Mozafari

Abstract

Building retrieval-augmented generation (RAG) systems often requires combining separate tools for retrieval, re-ranking, and generation, with incompatible data formats, evaluation pipelines, and deployment workflows. We present \framework, an open-source Python toolkit that unifies these stages in a single modular framework.\footnote{PyPI: \url{https://pypi.org/project/rankify/}} \textsuperscript{,}\footnote{GitHub: \url{https://github.com/DataScienceUIBK/Rankify}} \textsuperscript{,}\footnote{Docs: \url{https://rankify.readthedocs.io}}% \textsuperscript{,}\footnote{Video: \url{https://youtu.be/kkLzomrM2ec}} \framework provides 42 benchmark datasets with pre-retrieved documents and pre-built indices, 15 retrievers (sparse, dense, and reasoning-augmented), and 24 re-ranking models spanning 41 pointwise, pairwise, and listwise variants. It also supports \textbf{6} RAG strategies across four inference backends (Hugging Face, vLLM, LiteLLM, and OpenAI), enabling consistent experimentation from local models to hosted APIs. A unified pipeline interface allows users to compose retrieve--rerank--generate workflows in a few lines of code, while an agentic assistant (\texttt{RankifyAgent}), a REST server (\texttt{RankifyServer}), and an interactive web playground support deployment and non-programmatic exploration. Across 200+ configurations on QA and BEIR/TREC benchmarks with six generator LLMs, re-ranking consistently improves downstream performance, yielding gains of 5--15 points in Exact Match and up to 8.5 points in RAGAS context precision across diverse retriever--generator combinations.

Information Retrieval and Search BehaviorTopic ModelingNatural Language Processing Techniquesretrieval-augmented generationre-rankingdocument retrieval

More by Abdelrahman Abdallah