Talk · Conference on Empirical Methods in Natural Language Processing · January 1, 2026

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang, Rafiq Ali, Bdour Babillie

Abstract

Medical benchmarks are dominated by singleturn, multiple-choice clinical cases that poorly reflect real consultations.Practically, clinicians elicit evidence interactively and patient communication varies widely.We introduce MEDROUNDSQA, a multi-turn diagnostic benchmark derived from 1,387 boardexam cases across 17 specialties.Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed.We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis.Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points;(ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss. 1 Source QA Item (MCQ) A 64-year-old man presents with gradually increasing shortness of breath and a chronic dry cough.He has a 40 pack-year smoking history and drinks beer daily.His chest Xray reveals hyperinflated lungs.A. Pneumonia B. Emphysema (Answer Key …) C. Tuberculosis D. Heart failure Normalization & Extraction missing = NA; copy short phrases, enforce consistency Ground Truth Diagnosis: Emphysema Agent Final Diagnosis: Emphysema Question Choices 2 Canonical structured record (24-slot schema) 3 Full Multi-turn Dialog Doctor: Hello!How can I help you today?Patient: I've been getting more and more short of breath lately, and I've had this dry, hacking cough.Doctor: How long has this been going on?And how far can you walk before getting breathless?Patient: It's been going on for a couple of years now and seems to be getting worse.Now I get winded after just a few steps.I used to be able to walk for at least a few blocks without a problem.Doctor: Do you smoke or drink alcohol?Patient: Yes.I smoke about 1.5 packs of cigarettes a day and drink around four bottles of beer daily.Doctor: Your smoking history is significant.How long have you been smoking that amount?Patient: About 40 years.I started in my 20s and kept going ever since.Symptom descriptors Shortness of breath = Gradual Chronic dry cough = Yes Physical / exam findings Chest X-ray = Lung Hyperinflation Meta-info Specialty = Pulmonology History & context Smoking = 40 pack-years Alcohol = Beer daily

Artificial Intelligence in Healthcare and EducationSpeech and dialogue systemsClinical Reasoning and Diagnostic Skillsdiagnostic benchmarkpatient personasquestion difficulty

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Abstract

Related publications