The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by deco...
QoNext is a new framework that applies Quality of Experience (QoE) principles from networking and multimedia to evaluate foundation models in a conversational setting. It identifies experiential factors that influence user experience, collects human ratings in controlled interaction scenarios, and builds a database and neural predictor to estimate user satisfaction from system parameters. The framework demonstrates the ability to decode the mechanisms of user satisfaction and predict human sentiment across varied service conditions.
By Yijin Guo, Farong Wen, Ye Shen, Junying Wang, Qi Jia, Xiaohong Liu, Zicheng Zhang, Guangtao Zhai
The paper introduces UPHELD, a large benchmark of human-to-human dialogues written by professional script writers, featuring realistic turn densities and over 36,000 per-turn human annotations. It evaluates existing automatic metrics and LLM-as-a-judge methods, finding them unreliable against expert human judgment. Using UPHELD, the authors develop a Mixture-of-Judges framework that improves correlation with human assessments by about 30%.
By Ilija Subasic, Andrew Rabinovich, Zhao Chen
The paper evaluates whether breaking down evaluation tasks into simpler sub‑tasks improves the LLM-as-a-judge (LLMaJ) framework for reference‑free NLG assessment. Systematic comparisons across multiple datasets show no performance advantage for LLMaJ methods that use task decomposition over a fair baseline that does not. The authors attribute previously reported gains to the use of human labels for training rather than to decomposition itself, and note that LLMaJ without decomposition can match human annotators when such labels are available.
By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
By Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Bego\~na Altuna
SVG-Score introduces a human‑aligned evaluation framework for text‑to‑SVG generation, addressing the inadequacies of existing image‑based metrics like CLIPScore that poorly capture SVG‑specific errors such as color, count, and spatial inaccuracies. The authors first demonstrate that CLIP‑based scores are largely insensitive to these errors and that off‑the‑shelf Vision‑Language Models respond unevenly across error types and styles. They then create a human‑annotated Semantic Alignment dataset and develop two evaluators: a CLIP scorer adapted to vector graphics and a VLM judge trained with supervised fine‑tuning and reinforcement learning, enabling both fast large‑scale and expressive, interpretable assessment of SVG generators.