arXiv:2011.03783v3 Announce Type: replace-cross Abstract: In this work, we introduce the construction of a machine translation (MT) assisted and human-in-the-loop multilingual parallel corpus with an...
Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.
arXiv:2607.00890v2 Announce Type: replace Abstract: Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synth...
EnSiTa is a trilingual multi‑domain parallel dataset and benchmark for English, Sinhala, and Tamil. It contains human post‑edited training data across seven domains and professionally translated test sets for those domains plus an additional one, all produced through a multi‑year, rigorously quality‑controlled process. The authors use EnSiTa to conduct a comprehensive study of domain‑specific machine translation across six language directions, comparing from‑scratch Transformers, pre‑trained models, and decoder‑only LLMs under various training‑data sizes, model scales, and domain settings.
arXiv:2607. 20241v1 Announce Type: cross Abstract: Culturally loaded translation poses unique challenges for machine translation (MT), as meanings are deeply embedded in socio-cultural contexts beyond surface linguistic forms.
The paper investigates how to fairly compare language models across languages, noting that current evaluation methods vary widely and lack empirical validation. By training controlled monolingual models on parallel data and testing multilingual LLMs, the authors find that many normalized metrics suffer from biases due to tokenization, encoding, and orthographic differences. Instead, they recommend using sentence‑level negative log‑likelihood over semantically equivalent sequences for more reliable cross‑lingual comparisons.
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
The paper audits 51 recent multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented; low‑resource languages face a task equity gap. Representativeness is compromised by heavy reliance on English translations, and rigor is weakened by benchmark contamination and parallel‑corpus overlap. The authors propose concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination‑aware, and community‑aligned.
Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).
The study evaluates Arabic–Russian machine translation by comparing seven fine‑tuned neural machine translation (NMT) models with four few‑shot large language models (LLMs) on a new 15.47 million‑pair corpus split into 20k/5k/5k. Fine‑tuned NLLB‑1.3B achieves the best performance (BLEU 16.3, COMET 0.738), while the best few‑shot LLM, Aya‑Expanse 8B, scores only BLEU 1.7 on 500 sentences. Error analysis shows that low lexical overlap between Arabic and Russian is the main source of failures, and statistical tests confirm significant performance gaps between most models.
arXiv:2609.23490v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, curr...
The paper reports UvA-MT’s submission to the WMT 2026 General Machine Translation shared task, competing in the unconstrained track for all 23 translation directions. The approach leverages test‑time methods of Large Language Models, using sequential sampling with a refinement prompt to generate multiple translation candidates and then employing LLMs as pairwise judges in a round‑robin voting scheme to select the best candidate. Offline evaluation on the WMT25 benchmark shows a clear improvement over traditional point‑wise best‑of‑N selection.