KULTURE Bench is a new evaluation framework created to assess language models within the Korean cultural context. It includes datasets of cultural news, idioms, and poetry and tests models at word, sentence, and paragraph levels for cultural comprehension and reasoning. Experiments with models trained on different corpora show that current models still lack deep understanding of Korean cultural texts.
By Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, Hansaem Kim
The paper introduces TriviaRoomQA, a multilingual benchmark of 3,300 parallel multiple-choice questions across 288 topics, plus an additional 5,340 French-only questions. It evaluates 30 open-weight large language models (7–70 B parameters) from various regions, finding strong performance on knowledge-intensive subjects like history and geography but weaker results on everyday popular‑culture topics such as celebrities, music, movies, and news. The study also shows that model performance varies across languages for the same questions, indicating that factual knowledge access is not always language‑independent.
By Anna Mosolova, Djam\'e Seddah
KLEAR is a framework that creates knowledge‑matched contrasts for Korean LLM evaluation by varying application demand while keeping the required knowledge constant. It identifies the knowledge behind each source question, generates more complex application questions, and refines them through automated and human validation. Experiments show that these controlled sets reveal persistent failures in LLMs when application demand increases, even with the necessary knowledge supplied.
By Seoyeon Kim, Jeonghyun Kang, Hamin Koo, Barom Kang, Heondeuk Lee, Jaehyung Kim
Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics.
arXiv:2607. 06327v1 Announce Type: cross Abstract: Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English.
By Andrea Alfarano, Andrea Bacciu, Saab Mansour, Amin Mantrach, Marcello Federico
The paper introduces the concept of Language Specific Knowledge (LSK), showing that multilingual language models can answer certain queries better when prompted in a language other than English, sometimes even in low‑resource languages. It defines a language‑selection problem and presents several baseline methods, including the authors’ LSKExtractor, to empirically demonstrate that choosing the optimal language can improve question‑answering performance across datasets covering cultural and social norms. Experiments reveal non‑intuitive mappings, such as Gemma models excelling on Chinese and Middle Eastern topics in Spanish and Qwen models performing best on authority and responsibility queries in Arabic and Chinese.
By Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-T\"ur
The study investigates whether large language models (LLMs) are more prone to errors when they doubt the plausibility of input data, a phenomenon termed context‑memory conflict. Using non‑English and low‑resource language datasets, the authors generate text from factual, counterfactual, and fictional RDF triples in English, Czech, Slovak, and Upper Sorbian, and evaluate faithfulness with both human annotations and an LLM judge (Kimi K3). Contrary to expectations, the results show only a weak context‑memory conflict: counterfactual inputs receive slightly lower faithfulness scores than factual ones, and the choice of LLM judge can significantly affect perceived conflict strength.
By Peter Kochelka, Ale\v{s} Manuel Pap\'a\v{c}ek, Vojt\v{e}ch Dvo\v{r}\'ak, Ond\v{r}ej Du\v{s}ek
arXiv:2608. 15535v1 Announce Type: cross Abstract: We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs).
By Rinit Jain, Tirthraj Mahajan, Advait Joshi, Raviraj Joshi
arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.
By Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Shaoxiong Ji, Hassan Alhuzali, Yuechen Jiang, Jimin Huang, Sophia Ananiadou
arXiv:2605.26937v3 Announce Type: replace-cross
Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge ben...
By Luca Giordano, Simon Razniewski
SWORD is a new benchmark that tests large language models’ ability to reject factually incorrect statements across eight major languages by distorting Wikidata triples. The benchmark reveals that models often perform better on semantically plausible distortions than on random ones, indicating a reliance on distributional familiarity rather than true factual verification. It also shows significant performance drops for East Asian languages, with gaps up to 28 percentage points, highlighting asymmetric multilingual factual reasoning capabilities.
By Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun
Systematically evaluating the factuality of large language models with the FACTS Benchmark Suite.