The paper introduces CSM-MTBench, a benchmark for evaluating machine translation on Chinese social media text. It addresses two main challenges: limited parallel data due to slang and stylistic nuances, and inadequate evaluation metrics that miss these informal features. The benchmark includes two expert-curated subsets—Fun Posts and Social Snippets—and proposes specialized evaluation methods for each, revealing significant differences among over 20 MT models in handling semantic and stylistic aspects.
By Kaiyan Zhao, Zheyong Xie, Zhongtao Miao, Xinze Lyu, Yao Hu, Shaosheng Cao
arXiv:2607. 20241v1 Announce Type: cross Abstract: Culturally loaded translation poses unique challenges for machine translation (MT), as meanings are deeply embedded in socio-cultural contexts beyond surface linguistic forms.
By Yiming Wang, Jiayuan Di
arXiv:2601. 16217v2 Announce Type: replace-cross Abstract: Large language models increasingly mediate multilingual professional communication, where useful generation requires adapting to community conventions about which expressions are retained, translated, or mixed.
By Qingyan Yang, Tongxi Wang, Yunsheng Luo
The paper introduces a three‑layer checklist-and-judge framework to evaluate interpreter agents that mediate live conversation across languages. It assesses semantic, pragmatic, and cultural‑social dimensions—naturalness, intent, and social appropriateness—rather than just fidelity, in both single‑turn and multi‑turn settings. Extensive validation shows that conventional MT metrics miss failures in stronger interpreters, and that context, structured instructions, and cultural cues influence communicative success.
By Faiz Ghifari Haznitrama, Alice Oh
The paper introduces Retrieval-Enhanced Cultural Evaluation (CUE-MT), a framework that trains social media translation models by incorporating a calibrated, domain‑specific Judger and an Agent that retrieves external evidence to generate concise hints for scoring translations. Using this approach, CUE-MT outperforms comparable open‑source baselines across multiple model sizes on both CUE‑MT‑Bench and three public benchmarks, achieving cultural effectiveness gains of 7.44–15.12 percentage points in human evaluation. Retrieval notably boosts the Judger’s four‑way classification accuracy by 8.07 percentage points compared to a no‑retrieval baseline, and ablation studies confirm the benefits of hard‑example mining and Agent‑enhanced rewards.
By Linjuan Wu, Yiwen Qiu, Xinze Lyu, Dingming Li, Hui Yao, Zhongyi Sang, Ye Guo, Daoxin Zhang, Zhe Xu, Fei zhao, Mu Chuan, Yao Hu, Weiming Lu, Yongliang Shen
Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present T...