Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).
By Hala Almaghout, Christian Federmann, Qin Gao
arXiv:2610.19166v1 Announce Type: new
Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that co...
By Anubhav Jangra, Adrian de Wynter
The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by deco...
QoNext is a new framework that applies Quality of Experience (QoE) principles from networking and multimedia to evaluate foundation models in a conversational setting. It identifies experiential factors that influence user experience, collects human ratings in controlled interaction scenarios, and builds a database and neural predictor to estimate user satisfaction from system parameters. The framework demonstrates the ability to decode the mechanisms of user satisfaction and predict human sentiment across varied service conditions.
By Yijin Guo, Farong Wen, Ye Shen, Junying Wang, Qi Jia, Xiaohong Liu, Zicheng Zhang, Guangtao Zhai
The paper introduces UPHELD, a large benchmark of human-to-human dialogues written by professional script writers, featuring realistic turn densities and over 36,000 per-turn human annotations. It evaluates existing automatic metrics and LLM-as-a-judge methods, finding them unreliable against expert human judgment. Using UPHELD, the authors develop a Mixture-of-Judges framework that improves correlation with human assessments by about 30%.
By Ilija Subasic, Andrew Rabinovich, Zhao Chen
The paper evaluates whether breaking down evaluation tasks into simpler sub‑tasks improves the LLM-as-a-judge (LLMaJ) framework for reference‑free NLG assessment. Systematic comparisons across multiple datasets show no performance advantage for LLMaJ methods that use task decomposition over a fair baseline that does not. The authors attribute previously reported gains to the use of human labels for training rather than to decomposition itself, and note that LLMaJ without decomposition can match human annotators when such labels are available.
By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
The paper evaluates recent large language models (LLMs) for knowledge graph-to-text generation against traditional fine‑tuned encoder‑decoder models. Using both reference‑based metrics and graph‑grounded, reference‑free LLM‑based evaluation, the authors find that LLMs close the performance gap on reference metrics and outperform fine‑tuned models on graph‑grounded metrics. The study also shows that LLM‑based evaluation scores correlate poorly with reference metrics, indicating they assess different aspects of quality, and that prompted LLMs produce fewer contradictions and omissions, especially as graph size increases.
By Priyanka, Renaud Delbru, Souhail Bakkali, Micka\"el Coustaty, Antoine Doucet
The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
By Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Bego\~na Altuna
The paper evaluates recent large language models (LLMs) for knowledge graph‑to‑text generation against traditional fine‑tuned encoder‑decoder models. Using both reference‑based metrics and graph‑grounded, reference‑free LLM‑based evaluation, the study finds that LLMs largely close the performance gap on reference metrics and outperform fine‑tuned models on graph‑grounded metrics. The results also show that LLM‑based evaluation correlates weakly with reference metrics, indicating that each captures distinct aspects of generation quality, and that prompted LLMs produce fewer contradictions and omissions, especially as graph size grows.
arXiv:2504.11582v3 Announce Type: replace
Abstract: How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection...
By Dayeon Ki, Kevin Duh, Marine Carpuat
SVG-Score introduces a human‑aligned evaluation framework for text‑to‑SVG generation, addressing the inadequacies of existing image‑based metrics like CLIPScore that poorly capture SVG‑specific errors such as color, count, and spatial inaccuracies. The authors first demonstrate that CLIP‑based scores are largely insensitive to these errors and that off‑the‑shelf Vision‑Language Models respond unevenly across error types and styles. They then create a human‑annotated Semantic Alignment dataset and develop two evaluators: a CLIP scorer adapted to vector graphics and a VLM judge trained with supervised fine‑tuning and reinforcement learning, enabling both fast large‑scale and expressive, interpretable assessment of SVG generators.
The paper introduces Iterative MBR Distillation for Error Span Detection (ESD) in machine translation, a self‑evolution framework that replaces human annotations with pseudo‑labels generated by a large language model. By iteratively applying Minimum Bayes Risk decoding, the method produces high‑quality error spans without costly human effort. Experiments on WMT Metrics Shared Task datasets show that models trained solely on these pseudo‑labels outperform both unadapted baselines and supervised models trained on human data at system and span levels, while keeping sentence‑level performance competitive.
By Boxuan Lyu, Haiyue Song, Zhi Qu