arXiv:2509. 07829v4 Announce Type: replace-cross Abstract: Literary translation has recently gained attention as a distinct and complex task in machine translation research, yet translation by small open models remains an open problem, particularly for low-resource languages such as Romanian.
The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
The paper presents a retrieval‑augmented generation (RAG) system combined with prompt engineering for Japanese‑to‑Chinese translation, focusing on sentences with noun‑modifying clause constructions (NMCCs). It integrates linguistic analysis, embedding‑based retrieval, and prompt construction before generating translations with GPT‑4o, without altering the base model. Experiments on a 66‑sentence test set show that increasing the knowledge‑base size from 0 to 2,000 examples raises macro‑averaged BLEU scores from 24.28 to 29.96, indicating improved translation quality for NMCC‑rich sentences.
arXiv:2606. 05924v1 Announce Type: cross Abstract: Literary translation poses unique challenges due to the scarcity of high-quality annotated data and the need to balance expression fluency with literary effect.
arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating l...
Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present T...
The paper proposes a new way to evaluate language models on cultural understanding by focusing on interpretive depth rather than just factual recall. It argues that literary interpretation, where scholars can disagree yet still assess the quality of evidence, provides a useful framework for testing how models handle cultural references, reuse, and transformation across texts. The authors suggest linking evidence-centered benchmarks, preserving scholarly disagreement, and conducting model-development experiments on literary data, with Danish literature as a starting point for broader applications.
EviSI is an evidence‑based evaluation agent for low‑latency simultaneous speech‑to‑speech translation. It combines Multidimensional Quality Metrics with interpreter‑developed criteria, using shared source evidence to assess four dimensions—Anchor, Event, Logic, and Fluency—while deduplicating verified errors before scoring. On English‑to‑Chinese and Chinese‑to‑English data, EviSI’s rankings correlate strongly with human judgments, outperforming BLEU and COMET, and its multilingual extension maintains these correlations across five language directions.
The paper explores how to improve literary machine translation by using datasets that contain multiple valid translations of the same source text. It introduces a filtering framework that selects source texts whose references show meaningful variation while staying faithful, based on semantic similarity. Experiments show that fine‑tuning on medium to high similarity data outperforms low similarity data, and that using only this filtered subset can match or exceed performance achieved with the full unfiltered set. Additionally, the study compares synthetic translations generated by large language models with human expert translations, finding that fine‑tuning on human expert data yields better results in both automatic metrics and human evaluations, underscoring the continued importance of expert translations for literary MT.
arXiv:2609.14963v1 Announce Type: new Abstract: As large language models become capable translators of classical texts, a key challenge is deciding which outputs need expert review when no human refe...
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
The paper introduces VOLM, a framework that quantifies how much original value a human adds to a document beyond what a language model could generate from a task description alone. Unlike existing tools that focus on stylistic detection, VOLM extracts content at varying granularities, reconstructs it with an LLM, and compares these reconstructions to those derived from the task description. Evaluations across news articles, ICLR peer reviews, and argumentative essays show that VOLM can distinguish human-authored texts from LLM-generated ones while remaining robust to content-preserving transformations.