Underline Science Inc. · June 3, 2026
Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation
Association for Computational Linguistics 2026, Abdelrahman Abdallah, Andreas Herzinger, Jamie Holdcroft, Adam Jatowt, Jamshid Mozafari
Abstract
Building retrieval-augmented generation (RAG) systems often requires combining separate tools for retrieval, re-ranking, and generation, with incompatible data formats, evaluation pipelines, and deployment workflows. We present \framework, an open-source Python toolkit that unifies these stages in a single modular framework.\footnote{PyPI: \url{https://pypi.org/project/rankify/}} \textsuperscript{,}\footnote{GitHub: \url{https://github.com/DataScienceUIBK/Rankify}} \textsuperscript{,}\footnote{Docs: \url{https://rankify.readthedocs.io}}% \textsuperscript{,}\footnote{Video: \url{https://youtu.be/kkLzomrM2ec}} \framework provides 42 benchmark datasets with pre-retrieved documents and pre-built indices, 15 retrievers (sparse, dense, and reasoning-augmented), and 24 re-ranking models spanning 41 pointwise, pairwise, and listwise variants. It also supports \textbf{6} RAG strategies across four inference backends (Hugging Face, vLLM, LiteLLM, and OpenAI), enabling consistent experimentation from local models to hosted APIs. A unified pipeline interface allows users to compose retrieve--rerank--generate workflows in a few lines of code, while an agentic assistant (\texttt{RankifyAgent}), a REST server (\texttt{RankifyServer}), and an interactive web playground support deployment and non-programmatic exploration. Across 200+ configurations on QA and BEIR/TREC benchmarks with six generator LLMs, re-ranking consistently improves downstream performance, yielding gains of 5--15 points in Exact Match and up to 8.5 points in RAGAS context precision across diverse retriever--generator combinations.
More by Abdelrahman Abdallah
- A Paired Printed-Handwritten Arabic Line Dataset for Handwriting Recognition and Cross-Modal RetrievalScientific Data · 2026
- A multilingual Arabic and English receipt dataset for text detection, recognition and information extractionZenodo · 2026
- UIBK-Embedding: Effective Text Retrieval by Unifying Dense and Late-Interaction RetrievalarXiv · 2026
- MemRevise: A Self-Revising Memory for Conversational SearcharXiv · 2026
