KLEAR is a framework that creates knowledge‑matched contrasts for Korean LLM evaluation by varying application demand while keeping the required knowledge constant. It identifies the knowledge behind each source question, generates more complex application questions, and refines them through automated and human validation. Experiments show that these controlled sets reveal persistent failures in LLMs when application demand increases, even with the necessary knowledge supplied.
arXiv:2510.18368v2 Announce Type: replace Abstract: We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cu...
The paper introduces KBS, a Korean Benchmark Suite that includes six datasets—Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval—covering nine capabilities across 9,396 items. It emphasizes a non‑translation approach by reviewing, localizing, and redesigning tasks for Korean, and evaluates Korean‑developed, multilingual open, and closed API models under protocols that consider model access, prompt format, and scoring differences. The results highlight that Korean benchmark scores are influenced by task construction, scoring rules, prompting, and post‑training, suggesting that evaluations should report both final scores and the settings used.
arXiv:2605. 26937v2 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood.
arXiv:2606. 27047v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge.
arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.
arXiv:2605.26937v3 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge ben...
The paper introduces a knowledge‑graph‑based evaluation framework, S3KG, to assess whether large language models truly understand context in question answering tasks. S3KG combines structural and semantic signals into a single similarity score and is paired with a diagnostic analysis that pinpoints reasoning errors at the triplet level. Across nine benchmarks, the method outperforms existing baselines, achieving up to +7.6 F1 points and an AUROC of 0.973.
The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.
Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured...
arXiv:2511.07910v3 Announce Type: replace Abstract: Large Language Models (LLMs) achieve excellent performance in natural language reasoning tasks through pre-training on vast unstructured text, enab...
arXiv:2603. 02830v2 Announce Type: replace-cross Abstract: Predicting future student responses to questions is particularly valuable for educational learning platforms where it enables effective interventions.
The paper introduces a knowledge localization framework for mixture-of-experts large language models that exploits cross-lingual inconsistencies in factual recall. By comparing routing decisions for languages where the model succeeds versus fails on difficult factual questions, the method identifies a small set of experts crucial for answering specific knowledge queries. Deactivating roughly 20 of 6,000 experts often causes the model to fail in over 40% of cases, demonstrating the approach’s effectiveness and scalability.