Talk · January 1, 2026
Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts
Meruyert Aristombayeva, Jason S Lucas, Chaewan Chun, Dongwon Lee
Abstract
While text-based hallucination detection is well studied, reference-free detection of factual alterations in speech remains underexplored, especially for low-resource languages.Our spoken benchmark comprises 12,013 English, Russian, and Kazakh news samples with three synthetic alteration types and three severity levels, pairing source articles with rewrites as text, synthesized audio, and ASR transcripts.We add 290 fact-checked misinformation items collected in Russian (225) and Kazakh (65), translated into the other language and rendered through the same TTS-ASR pipeline.We evaluate finetuned multilingual encoders and zero-shot multimodal decoders on text, transcripts, and audio.Detectors receive only target inputs without source articles or external evidence; the task evaluates reference-free classification rather than evidence-grounded verification.Encoder degradation from source text to transcripts generally tracks per-language ASR error on the binary task.Among decoders, only Gemma-3n exceeds the binary majority-class baseline in macro-F1, on transcripts only; the other four do not exceed their respective baselines.Comparisons between audio and transcripts are confounded by differences in evaluation coverage and class balance.Synthetic-trained detectors achieve 0.82-0.88macro-F1 on real-world misinformation source text; Russian provenance analysis reveals veracity-related and modeldependent machine-style signals, a key confound in synthetic hallucination benchmarks.
Related publications
- TRILOGUE: A Trilingual Spoken Dialogue Fact-Checking Benchmark with Evidence and Paired AudioFindings of the Association for Computational Linguistics: EMNLP · 2026 · Same author: Meruyert Aristombayeva
- Guiding the uncharted: the emerging (and missing) policies on Generative AI in higher educationFrontiers in Education · 2025 · Citations: 9 · Same author: Meruyert Aristombayeva
- Optimizing the Mean Shift Algorithm for Efficient ClusteringMathematics · 2025 · Citations: 4 · Same author: Meruyert Aristombayeva
- FROM AN ANNOTATED CORPUS TO AN ONTOLOGY: A FORMAL MODEL OF MULTI-LEVEL SEMANTIC ANNOTATION OF FAKE NEWS IN THE KAZAKH–RUSSIAN MEDIA SPACE (KazFakeCorpus / KazFakeOnto)Russian Digital Libraries Journal · 2026 · Shared topic: Misinformation and Its Impacts
