Talk · Conference on Empirical Methods in Natural Language Processing: System Demonstrations · January 1, 2026
HintEval: An Open-Source Python Toolkit for Hint Generation and Hint Evaluation
Jamshid Mozafari, Bhawna Piryani, Abdelrahman Abdallah, Adam Jatowt
Abstract
Large Language Models (LLMs) increasingly provide direct answers to user questions, raising concerns about reduced engagement in critical thinking and problem-solving.Hint generation offers an alternative by guiding users toward answers without revealing them, while hint evaluation assesses the quality of such guidance.Research in this area is hindered by fragmented datasets, inconsistent annotation formats, and evaluation tools that are often dataset-specific or unavailable.To address these challenges, we introduce HINTEVAL 1 , an open-source Python library 2 for unified hint generation and evaluation.HINTEVAL standardizes access to diverse hint datasets, supports answer-aware and answer-agnostic generation methods, and implements multiple evaluation metrics within a shared data model.The toolkit enables reproducible experimentation, cross-dataset analysis, and multi-dimensional evaluation with minimal engineering effort.We further demonstrate its utility through human studies in which participants assess generated hints and use them to answer questions, showing that hints can effectively support users in reaching correct answers.HINTEVAL is accompanied by comprehensive documentation 3 , an executable Google Colab notebook for rapid experimentation, and a demonstration video 4 .By promoting consistent evaluation practices and lowering barriers to entry, it facilitates systematic research on hint-based question answering (QA) in NLP and IR.
More by Abdelrahman Abdallah
- A Paired Printed-Handwritten Arabic Line Dataset for Handwriting Recognition and Cross-Modal RetrievalScientific Data · 2026
- A multilingual Arabic and English receipt dataset for text detection, recognition and information extractionZenodo · 2026
- UIBK-Embedding: Effective Text Retrieval by Unifying Dense and Late-Interaction RetrievalarXiv · 2026
- MemRevise: A Self-Revising Memory for Conversational SearcharXiv · 2026
