The paper introduces a new corpus of 1,101 human simultaneous interpreting segments, each scored on meaning transfer, delivery quality, and perceived latency. It demonstrates that standard large‑language‑model prompting and scalar supervision fail to align with these rubric dimensions, producing near‑zero correlation with human ratings. By adding dual regression heads to a LoRA‑adapted COMET‑KIWI encoder, the authors achieve modest Pearson correlations (0.388 for meaning transfer and 0.301 for delivery quality) on a held‑out test set, improving over the frozen baseline.
arXiv:2601. 18253v2 Announce Type: replace-cross Abstract: Accurate evaluation of user satisfaction is critical for iterative development of conversational AI.
By Peng Sun, Xiangyu Zhang, Duan Wu, Lu Tan, Jian Lin, He Yang, Qi Qian, Yikai Wang
EviSI is an evidence‑based evaluation agent for low‑latency simultaneous speech‑to‑speech translation. It combines Multidimensional Quality Metrics with interpreter‑developed criteria, using shared source evidence to assess four dimensions—Anchor, Event, Logic, and Fluency—while deduplicating verified errors before scoring. On English‑to‑Chinese and Chinese‑to‑English data, EviSI’s rankings correlate strongly with human judgments, outperforming BLEU and COMET, and its multilingual extension maintains these correlations across five language directions.
By Ben Yan, Zongyao Li, Xiaoyu Chen, Daimeng Wei, Weidong Liu, Huan Zhao, Chong Li, Yaode Wang, Yuzhe Shang
arXiv:2609.22778v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require exper...
By Yuhan Lu, Yi Yao, Hua Shen, Katie Aafjes-van Doorn, Zhaonan Wang
arXiv:2601.08654v3 Announce Type: replace
Abstract: Rubric-based text evaluation increasingly relies on large language models (LLMs) as scalable judges, yet frozen black-box models can interpret the...
By Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, Yushun Dong
arXiv:2607. 12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality.
By Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran