← Back to all news
arXiv Computation and Language By Seoyeon Kim, Jeonghyun Kang, Hamin Koo, Barom Kang, Heondeuk Lee, Jaehyung Kim

KLEAR: Knowledge-Matched LLM Evaluation in Korean across Application Requirements via Controlled Benchmark Transformation

Read the original on arXiv Computation and Language →

KLEAR is a framework that creates knowledge‑matched contrasts for Korean LLM evaluation by varying application demand while keeping the required knowledge constant. It identifies the knowledge behind each source question, generates more complex application questions, and refines them through automated and human validation. Experiments show that these controlled sets reveal persistent failures in LLMs when application demand increases, even with the necessary knowledge supplied.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Oct 16

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

The paper introduces KBS, a Korean Benchmark Suite that includes six datasets—Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval—covering nine capabilities across 9,396 items. It emphasizes a non‑translation approach by reviewing, localizing, and redesigning tasks for Korean, and evaluates Korean‑developed, multilingual open, and closed API models under protocols that consider model access, prompt format, and scoring differences. The results highlight that Korean benchmark scores are influenced by task construction, scoring rules, prompting, and post‑training, suggesting that evaluations should report both final scores and the settings used.

By Yeonkyoung So, Jongmin Kim, Sungmok Jung, Gyuseong Lee, Sangho Kim, Jongyeon Park, Joonhak Lee, Seho Pyo, Gyeongje Cho, Seorin Kim, Jisoo Kim, Suyoung Park, Hyunji M. Park, Yelim Ahn, Yeongho Seo, Jaejin Lee
llmsbenchmarks
More like this →
arXiv AI
Jun 10

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.

By Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun
llmsbenchmarks
More like this →