arXiv Computation and Language
KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Sep 18
KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms
KoNeoBench is a curated dataset designed to evaluate large language models’ understanding of Korean neologisms. It contains 1,785 recently attested Korean words from online news since 2020, each accompanied by usage examples, word‑formation analyses, and dictionary‑style definitions. The authors define four evaluation tasks, report results from recent models and a human baseline, and find that current LLMs struggle with recovering source components, distinguishing semantic categories, and generating accurate definitions.
arXiv Computation and Language
Aug 31
CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms
CNeo-Bench is a new benchmark comprising 4,759 Chinese neologisms, each with reference definitions and categorized by linguistic mechanisms such as phonetic substitution and visual character decomposition. The benchmark includes a two-tier evaluation framework that tests whether models can describe a neologism and whether they can manipulate its underlying mechanism. Evaluation of 18 large language models shows that most perform poorly on definition generation (below 40%) and exhibit a recognition‑manipulation gap, often paraphrasing rather than restoring the original form; few‑shot prompting helps but does not fully resolve the errors.
arXiv Machine Learning
Aug 4
Cultural Awareness is Represented but Not Decoded: Tracing Mythological Knowledge across 18 Open-Source LLMs
arXiv:2608. 02486v1 Announce Type: cross Abstract: Open-source LLMs reliably name Zeus, Jupiter, and Thor, but recover their counterparts in less-represented traditions like Finnish, Slavic, Egyptian, or Chinese mythology far less consistently.
arXiv AI
Oct 15
North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
arXiv:2610.15282v1 Announce Type: cross Abstract: Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolin...
arXiv Computation and Language
Sep 10
Do LLMs Make More Mistakes If They Do Not Believe the Input Data?
The study investigates whether large language models (LLMs) are more prone to errors when they doubt the plausibility of input data, a phenomenon termed context‑memory conflict. Using non‑English and low‑resource language datasets, the authors generate text from factual, counterfactual, and fictional RDF triples in English, Czech, Slovak, and Upper Sorbian, and evaluate faithfulness with both human annotations and an LLM judge (Kimi K3). Contrary to expectations, the results show only a weak context‑memory conflict: counterfactual inputs receive slightly lower faithfulness scores than factual ones, and the choice of LLM judge can significantly affect perceived conflict strength.
arXiv Computation and Language
Sep 18
LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations
LMEnt is a toolkit designed to study how language models acquire and represent world knowledge. It includes a richly annotated pretraining corpus, an improved entity-based retrieval method, and twelve pretrained models with numerous checkpoints. The suite enables controlled experiments linking entity mentions in training data to downstream knowledge performance, revealing how co-occurrence patterns influence learning and editing difficulty.