arXiv:2608.28405v1 Announce Type: new
Abstract: Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common...
By Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan, Bich Ngoc Doan, Jia Wang Peh, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Zhengyuan Liu, JinYeong Bak, Wafi Shamdi, Soo Kai Chie, Liew Yu Siong, Aina Azyyati Binti Mohamad Rezal, Lew Yan Yan Vanessa, Huadan Wu, Dylan Raharja, Nadya Yuki Wangsajaya, Akane Fukushige, Kazushi Kato, Koji Inoue, Tatsuya Kawahara, Jaehyung Seo, Dongjun Kim, Seungyoon Lee, Zi Haur Pang, Rui Yang Tan, Charibeth Ko Cheng, Maria Regina Justina Estuar, Jann Railey Montalan, Pham Minh Duc, Roy Ka-Wei Lee
The paper introduces MentorQA, a multilingual dataset and evaluation framework for mentorship-oriented question answering derived from long‑form videos. It contains nearly 9,000 QA pairs across four languages and defines evaluation dimensions such as clarity, alignment, and learning value that extend beyond factual accuracy. Experiments show that Multi‑Agent QA pipelines outperform other architectures, especially on complex topics and low‑resource languages, while automated LLM‑based evaluation shows variable alignment with human judgments.
By Parth Bhalerao, Diola Dsouza, Ruiwen Guan, Oana Ignat
The paper investigates cultural biases in large language models (LLMs) by introducing the Culture-Related Open Questions (CROQ) dataset, which contains 24‑language questions about generic culture. Experiments reveal that LLMs disproportionately favor Japan in their responses, especially when prompted in high‑resource languages, while low‑resource languages tend to highlight countries where the language is official. The study also finds that these biases emerge after supervised fine‑tuning rather than during pre‑training.
By Joseba Fernandez de Landa, Carla Perez-Almendros, Jose Camacho-Collados
arXiv:2608. 15535v1 Announce Type: cross Abstract: We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs).
By Rinit Jain, Tirthraj Mahajan, Advait Joshi, Raviraj Joshi
arXiv:2509. 08022v3 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) with diverse human values is essential for safe and effective deployment, yet existing benchmarks often overlook cultural and demographic variation.
By Yao Liang, Dongcheng Zhao, Feifei Zhao, Guobin Shen, Yuwei Wang, Dongqi Liang, Yi Zeng
arXiv:2609.00565v1 Announce Type: cross
Abstract: Cultural fine-tuning has become the de facto paradigm for building culture-aware large language models (LLMs), yet existing optimization exclusively...
By Jingshen Zhang, Shaoyang Xu, Wenxuan Zhang