arXiv AI
North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 18
KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms
KoNeoBench is a curated dataset designed to evaluate large language models’ understanding of Korean neologisms. It contains 1,785 recently attested Korean words from online news since 2020, each accompanied by usage examples, word‑formation analyses, and dictionary‑style definitions. The authors define four evaluation tasks, report results from recent models and a human baseline, and find that current LLMs struggle with recovering source components, distinguishing semantic categories, and generating accurate definitions.
arXiv Computation and Language
Oct 12
Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border Communities
The paper investigates how large language models (LLMs) represent nine transborder ethnic communities along China’s southwestern border with mainland Southeast Asia, such as the Sami, Rohingya, and Kurds. It introduces the concept of identity ossification, where LLMs reduce fluid, cross-border identities to a single nation-state narrative, and presents the Transborder Identity Ossification Diagnostic Framework (TIODF) to differentiate between knowledge gaps and narrative framing. Experiments with GPT‑5.1 and DeepSeek‑V3.2 in English and Chinese show that ossification rates vary widely, with DeepSeek‑V3.2’s Chinese responses exhibiting the phenomenon in 62.6% of cases, indicating that narrative framing rather than mere knowledge absence drives the issue.
arXiv Computation and Language
Sep 14
KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
arXiv:2510.18368v2 Announce Type: replace Abstract: We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cu...
arXiv Computation and Language
Sep 2
Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
arXiv:2609.00565v1 Announce Type: cross Abstract: Cultural fine-tuning has become the de facto paradigm for building culture-aware large language models (LLMs), yet existing optimization exclusively...
arXiv AI
Aug 11
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
arXiv:2608. 09925v1 Announce Type: cross Abstract: Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts.
arXiv Computation and Language
Oct 15
KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?
arXiv:2610.19399v1 Announce Type: new Abstract: Korean neologisms arise through diverse mechanisms, a demanding test of language models beyond lexical memorization. To enable systematic evaluation, w...