Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).
By Hala Almaghout, Christian Federmann, Qin Gao
arXiv:2608.24327v2 Announce Type: replace
Abstract: With the advent of Large Language Models and its instruction following capabilities a promising application is the task of summarization. Within th...
By Enes Yavuz Ugan, Fabian Retkowski, Yuka Ko, Thai-Binh Nguyen, Maike Z\"ufle, Jan Niehues, Alexander Waibel
arXiv:2512. 06227v3 Announce Type: replace-cross Abstract: Real-world indicators play an important role in many Natural Language Processing (NLP) applications, such as life events for mental health analysis and risky behaviours for online safety, yet labelling such information is often costly and/or difficult due to its multi-label and dynamic nature.
By Junyu Mao, Anthony Hills, Talia Tseriotou, Maria Liakata, Aya Shamir, Dan Sayda, Dana Atzil-Slonim, Natalie Djohari, Pamela Ugwudike, Mahesan Niranjan, Stuart E. Middleton
The paper introduces Iterative MBR Distillation for Error Span Detection (ESD) in machine translation, a self‑evolution framework that replaces human annotations with pseudo‑labels generated by a large language model. By iteratively applying Minimum Bayes Risk decoding, the method produces high‑quality error spans without costly human effort. Experiments on WMT Metrics Shared Task datasets show that models trained solely on these pseudo‑labels outperform both unadapted baselines and supervised models trained on human data at system and span levels, while keeping sentence‑level performance competitive.
By Boxuan Lyu, Haiyue Song, Zhi Qu
arXiv:2609.21663v1 Announce Type: new
Abstract: Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equal...
By Hritika Sharma, Thibault Ba\~neras-Roux, Alessandra Pinto, Petr Motlicek, Hyunggu Jung, Esa\'u Villatoro-Tello, Somang Nam
PEDBench is a new benchmark for phonetic error detection (PED) that includes natural recordings of children’s speech errors, scripted error productions by actors, and synthetic error productions generated via phone‑conditioned text‑to‑speech. The benchmark provides high‑quality phonetic annotations verified by trained annotators. Evaluation of 13 state‑of‑the‑art speech models shows that they struggle to reliably detect phonetic errors, even in controlled conditions, and often misclassify correct phonemes as errors. Fine‑tuned speech‑language models perform better on word‑level PED, but a sim‑to‑real domain gap persists.
By Zhonghao Shi, Zackary Rackauckas, Jasmine Yip, Savannah Halpern, Simon Campos, Lauren Brace, Emma Yang, Tiantian Feng, Xuan Shi, Shrikanth Narayanan, Teresa Satterfield, Ying Xu
Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart.
arXiv:2609.09889v1 Announce Type: new
Abstract: Automatic Speech Recognition (ASR) technology is fundamental to customer service automation and large-scale transcription. However, even advanced ASR m...
By Yonghyun Jun, Jimin Lee, Hwan Chang, Dongho Shin, Seolah Kim, Hwanhee Lee
arXiv:2610.24809v2 Announce Type: replace
Abstract: Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during stre...
By Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi
arXiv:2601.16753v2 Announce Type: replace-cross
Abstract: Longitudinal information in radiology reports refers to the sequential tracking of findings across multiple examinations over time, which is...
By Xinyi Wang, Grazziela Figueredo, Ruizhe Li, Xin Chen
arXiv:2605. 20712v2 Announce Type: replace-cross Abstract: Automatic speech recognition replaces typing only when correction costs less than manual entry - a threshold determined by error types, not counts: fixing a misrecognized domain term costs far more than inserting a comma.
By Kavya Manohar, Arghya Bhattacharya, Kush Juvekar, Kumarmanas Nethil
arXiv:2606. 24825v1 Announce Type: cross Abstract: Part-of-Speech (POS) tagging is a foundational NLP task underpinning machine translation, information extraction, and syntactic parsing.
By Hariom Ingle, Ronit Ghode, Ishwari Gondkar, Jidnyasa Harad, Raviraj Joshi