Talk · Findings of the Association for Computational Linguistics: EMNLP · January 1, 2025
DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation
Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
Abstract
Large Language Models (LLMs) have transformed listwise document reranking by enabling global reasoning over candidate sets, yet single models often struggle to balance fine-grained relevance scoring with holistic cross-document analysis.We propose DeepAgentRank (DEAR), an open-source framework that decouples these tasks through a dual-stage approach, achieving superior accuracy and interpretability.In Stage 1, we distill token-level relevance signals from a frozen 13B LLaMA teacher into a compact {3, 8}B student model using a hybrid of cross-entropy, RankNet, and KL divergence losses, ensuring robust pointwise scoring.In Stage 2, we attach a second LoRA adapter and fine-tune on 20K GPT-4o-generated chain-of-thought permutations, enabling listwise reasoning with natural-language justifications.Evaluated on TREC-DL19/20, eight BEIR datasets, and NovelEval-2306, DEAR surpasses opensource baselines by +5.1 nDCG@5 on DL20 and achieves 90.97 nDCG@10 on NovelEval, outperforming GPT-4 by +3.09.Without fine-tuning on Wikipedia, DeAR also excels in open-domain QA, achieving 54.29 Top-1 accuracy on Natural Questions, surpassing baselines like MonoT5, UPR, and RankGPT.Ablations confirm that dual-loss distillation ensures stable calibration, making DEAR a highly effective and interpretable solution for modern reranking systems. 1 .
More by Abdelrahman Abdallah
- A Paired Printed-Handwritten Arabic Line Dataset for Handwriting Recognition and Cross-Modal RetrievalScientific Data · 2026
- A multilingual Arabic and English receipt dataset for text detection, recognition and information extractionZenodo · 2026
- UIBK-Embedding: Effective Text Retrieval by Unifying Dense and Late-Interaction RetrievalarXiv · 2026
- MemRevise: A Self-Revising Memory for Conversational SearcharXiv · 2026
