arXiv AI
Source-Free MT Evaluation Is Not MT Evaluation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
3d ago
In the Blind: Building Pseudo-References for MT Evaluation
arXiv:2609.13611v1 Announce Type: new Abstract: The WMT26 General MT task evaluates systems on 10 language pairs that have no human references (neither translated from scratch nor post-edited from MT...
arXiv AI
Sep 7
Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization
The paper introduces CAMS, a Claim‑Anchored Multi‑Document Summarization framework that decomposes source documents into atomic claims, resolves provenance deterministically from verbatim quotes to token spans, clusters equivalent claims across documents, and rewrites summaries so each sentence ends with claim identifiers linking back to source spans. CAMS separates provenance (an invariant for each emitted sentence) from faithfulness (an objective encouraged by selection, rewriting, and verification). Evaluations on MultiNews, DiverseSumm, and zero‑shot WCEP show that CAMS matches strong baselines in summary quality while improving faithfulness and citation precision, raising attribution accuracy from 38% to 64% and reducing human verification time per claim by 3.4×.
arXiv AI
Jun 24
Faithful by Construction: Claim-Anchored Attribution for Multi-Document Summarization
arXiv:2606. 23989v1 Announce Type: cross Abstract: End-to-end large language models (LLMs) produce fluent multi-document summaries but remain prone to hallucination, and the attributions they offer are typically coarse (whole documents or passages) and generated post hoc, leaving each summary statement hard to verify.
arXiv AI
3d ago
Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translation
arXiv:2609.14795v1 Announce Type: cross Abstract: There is a tradeoff in machine translation meta-evaluation between prioritizing alignment with adequacy versus fluency. The balance depends on the co...
arXiv Computation and Language
Sep 1
Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intent
The paper investigates how machine translation can be tailored to specific audiences and intents, a capability enabled by large language models (LLMs). By systematically evaluating purpose-driven MT across 50 languages, 5 model sizes, and 8 text domains, the authors find that explicit instructions significantly improve translation adaptiveness, especially for informal domains, larger models, and higher-resource languages. They also show that traditional MT metrics often penalize adapted translations and that models can self-generate useful instructions from context, closing a large portion of the adaptiveness gap.
arXiv AI
Sep 7
Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methods
The paper introduces behavioral correctness assumptions as a new framework for evaluating reference-based automatic evaluation methods in natural language generation. It defines a taxonomy of correctness-preserving and correctness-altering assumptions and operationalizes them through controlled response transformations to specify expected scoring behaviors. The authors evaluate a range of lexical, character-level, semantic, LLM-based, and hybrid evaluators, analyzing their behavior across multiple dimensions and finding that no evaluator satisfies all assumptions, revealing distinct behavioral trade-offs not evident from aggregate scores.