arXiv:2610.19399v1 Announce Type: new
Abstract: Korean neologisms arise through diverse mechanisms, a demanding test of language models beyond lexical memorization. To enable systematic evaluation, w...
By SungHo Kim, Eojin Jeon, SangKeun Lee
CNeo-Bench is a new benchmark comprising 4,759 Chinese neologisms, each with reference definitions and categorized by linguistic mechanisms such as phonetic substitution and visual character decomposition. The benchmark includes a two-tier evaluation framework that tests whether models can describe a neologism and whether they can manipulate its underlying mechanism. Evaluation of 18 large language models shows that most perform poorly on definition generation (below 40%) and exhibit a recognition‑manipulation gap, often paraphrasing rather than restoring the original form; few‑shot prompting helps but does not fully resolve the errors.
By Kaiyan Zhao, Zhongtao Miao, Zheyong Xie, Shaosheng Cao, Yoshimasa Tsuruoka
arXiv:2610.15282v1 Announce Type: cross
Abstract: Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolin...
By Kiwoong Park, Seyoung Song, Donghyun Woo, Alice Oh
The paper introduces “KoLing”, a bias‑controlled adversarial benchmark of 50.9k Korean instances designed to suppress superficial cues while preserving semantic content, and evaluates both LLM‑generated and LLM‑modified text detection. It also proposes “KoLingProbe”, an interpretable detector grounded in theoretical linguistics that models Korean‑specific properties across multiple linguistic levels. Experiments demonstrate that KoLingProbe outperforms strong baselines on all detection tasks and remains robust to adversarial attacks, with analyses showing that integrating complementary linguistic features across multiple levels improves both performance and interpretability.
By SungHo Kim, SangKeun Lee
KULTURE Bench is a new evaluation framework created to assess language models within the Korean cultural context. It includes datasets of cultural news, idioms, and poetry and tests models at word, sentence, and paragraph levels for cultural comprehension and reasoning. Experiments with models trained on different corpora show that current models still lack deep understanding of Korean cultural texts.
By Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, Hansaem Kim
The paper introduces CSM-MTBench, a benchmark for evaluating machine translation on Chinese social media text. It addresses two main challenges: limited parallel data due to slang and stylistic nuances, and inadequate evaluation metrics that miss these informal features. The benchmark includes two expert-curated subsets—Fun Posts and Social Snippets—and proposes specialized evaluation methods for each, revealing significant differences among over 20 MT models in handling semantic and stylistic aspects.
By Kaiyan Zhao, Zheyong Xie, Zhongtao Miao, Xinze Lyu, Yao Hu, Shaosheng Cao