The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2610.33749v1 Announce Type: new Abstract: As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We crit...
Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).
Qwen-MT-Lumen is an open‑source translation model submitted to the WMT 2026 General Machine Translation Shared Task. It employs a four‑stage pipeline that combines continued pre‑training, supervised fine‑tuning, online preference‑optimization with an LLM judge, and on‑policy distillation from a larger teacher to a Qwen3‑14B student. The system also uses structure‑aware inference and selective repair to preserve formatting and correct decoding errors, achieving an overall mean score of 4.60 on a 1–5 rubric and matching leading proprietary systems in both translation quality and instruction adherence.
Multi‑SFT is a new, commercially permissive dataset comprising 502,818 instruction‑response pairs in five European languages, created through translation and regeneration pipelines to address gaps where simple translation is insufficient. Human evaluation shows strong inter‑annotator agreement (≥0.87) across all languages. Fine‑tuning nine models from three families on Multi‑SFT yields performance competitive with larger, proprietary mixes and official instruction‑tuned releases, especially on safety‑related benchmarks.
arXiv:2609.23490v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, curr...
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.