The paper introduces the Last Translation Benchmark (LTB), a live dataset of human-authored and peer‑reviewed examples—including texts, images, audio, and videos—that are designed to break current state‑of‑the‑art machine translation models. Each example is accompanied by handcrafted verification rules that specify concrete failure cases, providing a reliable and actionable evaluation method. The benchmark aims to overcome the limitations of existing automatic metrics and gold human evaluations, which often lack reproducibility, objectivity, and scalability.
Machine-generated by The Flow from the publisher's headline and feed description
— not written or checked by a human. The full article lives at arXiv Computation and Language.
arXiv:2608. 15932v1 Announce Type: new Abstract: As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models of varying quality.
By William Kalikman, \v{S}imon Sukup, Michal Te\v{s}nar, Vil\'em Zouhar
As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models of varying quality. We propose augmenting existi...
Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).
TransClean introduces a benchmark for identifying and removing translation noise—unwanted text such as language labels, explanations, or bilingual repetitions—from large language model (LLM) outputs. The authors analyzed 790,000 translations from 12 LLMs across 22 language pairs, cataloguing 12 common noise patterns and creating 9,900 noisy‑clean pairs (8,800 synthetic, 1,100 authentic). They evaluated two extraction methods—a span‑based approach using quality estimation models and an LLM‑prompted method—demonstrating the first systematic framework to assess and improve translation cleanliness.
arXiv:2601.02933v4 Announce Type: replace
Abstract: Human evaluation is the gold standard for multilingual NLP, but is often skipped in practice and substituted with automatic metrics because it is n...
The paper reports UvA-MT’s submission to the WMT 2026 General Machine Translation shared task, competing in the unconstrained track for all 23 translation directions. The approach leverages test‑time methods of Large Language Models, using sequential sampling with a refinement prompt to generate multiple translation candidates and then employing LLMs as pairwise judges in a round‑robin voting scheme to select the best candidate. Offline evaluation on the WMT25 benchmark shows a clear improvement over traditional point‑wise best‑of‑N selection.
By Di Wu, Sergey Troshin, Wafaa Mohammed, Seth Aycock, Evgeniia Tokarchuk, Vlad Niculae, Christof Monz