arXiv:2609.13611v1 Announce Type: new Abstract: The WMT26 General MT task evaluates systems on 10 language pairs that have no human references (neither translated from scratch nor post-edited from MT...
The paper introduces CAMS, a Claim‑Anchored Multi‑Document Summarization framework that decomposes source documents into atomic claims, resolves provenance deterministically from verbatim quotes to token spans, clusters equivalent claims across documents, and rewrites summaries so each sentence ends with claim identifiers linking back to source spans. CAMS separates provenance (an invariant for each emitted sentence) from faithfulness (an objective encouraged by selection, rewriting, and verification). Evaluations on MultiNews, DiverseSumm, and zero‑shot WCEP show that CAMS matches strong baselines in summary quality while improving faithfulness and citation precision, raising attribution accuracy from 38% to 64% and reducing human verification time per claim by 3.4×.
arXiv:2606. 23989v1 Announce Type: cross Abstract: End-to-end large language models (LLMs) produce fluent multi-document summaries but remain prone to hallucination, and the attributions they offer are typically coarse (whole documents or passages) and generated post hoc, leaving each summary statement hard to verify.
arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author mean...
CiteVQA is a benchmark that evaluates multimodal large language models on both answering document questions and providing element‑level bounding‑box citations for the evidence used. It contains 1,897 questions across 711 PDFs in seven domains and two languages, with ground‑truth citations generated via an automated pipeline and strict quality control. The evaluation metric, Strict Attributed Accuracy (SAA), rewards a prediction only when both the answer and the cited region are correct, revealing widespread attribution hallucination in current models.
The paper introduces a reference‑free proxy‑judge framework for autoformalization, separating correctness into grounding, statement faithfulness, coverage, and inference. It presents three theoretical results linking proxy error rates to pass‑but‑wrong rates, the impact of false rejections on refinement, and a decomposition of final pass rates. The framework is instantiated with eight proxies and evaluated on ProofNet, miniF2F, e‑SNLI, and ProntoQA, showing that the predictions hold and identifying failure routes.
The paper addresses the disconnect between textual data quality (DQ) dimensions and practical DQ tools by making ISO/IEC 25024 and ISO/IEC 5259 metrics executable. It classifies metrics into generalizable, parameterized, and non-generalizable categories, and provides implementation guidelines for the first two. The authors implement these guidelines in the open-source library dqmeasure, demonstrating that the metrics respond predictably to injected errors, correlate with downstream machine learning performance, and scale linearly with dataset size.
arXiv:2609.14795v1 Announce Type: cross Abstract: There is a tradeoff in machine translation meta-evaluation between prioritizing alignment with adequacy versus fluency. The balance depends on the co...
The paper investigates how machine translation can be tailored to specific audiences and intents, a capability enabled by large language models (LLMs). By systematically evaluating purpose-driven MT across 50 languages, 5 model sizes, and 8 text domains, the authors find that explicit instructions significantly improve translation adaptiveness, especially for informal domains, larger models, and higher-resource languages. They also show that traditional MT metrics often penalize adapted translations and that models can self-generate useful instructions from context, closing a large portion of the adaptiveness gap.
The paper introduces behavioral correctness assumptions as a new framework for evaluating reference-based automatic evaluation methods in natural language generation. It defines a taxonomy of correctness-preserving and correctness-altering assumptions and operationalizes them through controlled response transformations to specify expected scoring behaviors. The authors evaluate a range of lexical, character-level, semantic, LLM-based, and hybrid evaluators, analyzing their behavior across multiple dimensions and finding that no evaluator satisfies all assumptions, revealing distinct behavioral trade-offs not evident from aggregate scores.
arXiv:2610.35261v1 Announce Type: new Abstract: Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whet...
arXiv:2608. 08283v1 Announce Type: cross Abstract: Although large language models can translate some historical languages surprisingly well, their usefulness in digital humanities workflows is limited by the lack of reliable evaluation.