KoNeoBench is a curated dataset designed to evaluate large language models’ understanding of Korean neologisms. It contains 1,785 recently attested Korean words from online news since 2020, each accompanied by usage examples, word‑formation analyses, and dictionary‑style definitions. The authors define four evaluation tasks, report results from recent models and a human baseline, and find that current LLMs struggle with recovering source components, distinguishing semantic categories, and generating accurate definitions.
arXiv:2610.19399v1 Announce Type: new Abstract: Korean neologisms arise through diverse mechanisms, a demanding test of language models beyond lexical memorization. To enable systematic evaluation, w...
CNeo-Bench is a new benchmark comprising 4,759 Chinese neologisms, each with reference definitions and categorized by linguistic mechanisms such as phonetic substitution and visual character decomposition. The benchmark includes a two-tier evaluation framework that tests whether models can describe a neologism and whether they can manipulate its underlying mechanism. Evaluation of 18 large language models shows that most perform poorly on definition generation (below 40%) and exhibit a recognition‑manipulation gap, often paraphrasing rather than restoring the original form; few‑shot prompting helps but does not fully resolve the errors.
arXiv:2610.15282v1 Announce Type: cross Abstract: Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolin...
The paper introduces “KoLing”, a bias‑controlled adversarial benchmark of 50.9k Korean instances designed to suppress superficial cues while preserving semantic content, and evaluates both LLM‑generated and LLM‑modified text detection. It also proposes “KoLingProbe”, an interpretable detector grounded in theoretical linguistics that models Korean‑specific properties across multiple linguistic levels. Experiments demonstrate that KoLingProbe outperforms strong baselines on all detection tasks and remains robust to adversarial attacks, with analyses showing that integrating complementary linguistic features across multiple levels improves both performance and interpretability.
KULTURE Bench is a new evaluation framework created to assess language models within the Korean cultural context. It includes datasets of cultural news, idioms, and poetry and tests models at word, sentence, and paragraph levels for cultural comprehension and reasoning. Experiments with models trained on different corpora show that current models still lack deep understanding of Korean cultural texts.
The paper introduces CSM-MTBench, a benchmark for evaluating machine translation on Chinese social media text. It addresses two main challenges: limited parallel data due to slang and stylistic nuances, and inadequate evaluation metrics that miss these informal features. The benchmark includes two expert-curated subsets—Fun Posts and Social Snippets—and proposes specialized evaluation methods for each, revealing significant differences among over 20 MT models in handling semantic and stylistic aspects.
arXiv:2510.18368v2 Announce Type: replace Abstract: We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cu...
arXiv:2608. 05850v1 Announce Type: cross Abstract: We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish.
arXiv:2604.19139v4 Announce Type: replace-cross Abstract: Repeated praise, canned reassurance, familiar contrasts, and conspicuous vocabulary are recurring subjects in discussions of large language m...
CHRONOBERG is a temporally structured corpus of English book texts covering 250 years, curated from Project Gutenberg and enriched with temporal annotations. It enables quantification of lexical semantic change via time‑sensitive Valence‑Arousal‑Dominance analysis and the creation of historically calibrated affective lexicons. Experiments show that language models trained sequentially on CHRONOBERG struggle to encode diachronic shifts, highlighting the need for temporally aware training and evaluation pipelines.
Chronoberg is a 250‑year English book corpus from Project Gutenberg, enriched with temporal annotations and affective lexicons (Valence‑Arousal‑Dominance). It allows researchers to track lexical semantic change over time and to evaluate how large language models handle sentiment and discriminatory language across different historical periods. The authors demonstrate that current LLMs struggle with diachronic shifts, underscoring the need for temporally aware training and evaluation pipelines.
We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling.