arXiv Computation and Language

Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts

The paper introduces MH‑INDIC, a framework that evaluates maternal‑health language model interactions in North Indian urban and semi‑urban settings by measuring cultural alignment across ten dimensions of reasoning. Using a survey of 102 pregnant and postpartum women, the authors assess ten LLMs, finding that while models approximate overall population‑level cultural patterns, they lack the demographic and household‑specific variation seen in humans. The study also demonstrates that human‑grounded conditioning improves both profile alignment and dialogue quality in generated interactions.

arXiv AI
2d ago

Whose Values, Which Contexts, and What Signals? Grounding Cultural Value Alignment with Re3Culture

arXiv:2610.41793v1 Announce Type: new Abstract: Cultural value alignment requires not only model optimization but also a well-specified target grounded in real-human evidence. Yet how such targets sh...

By Jing Yao, Hyunjin Hwang, Weihua Zheng, Bryan Chen Zhengyu Tan, Jia Wang Peh, Dylan Raharja, JinYeong Bak, Roy Ka-Wei Lee, Rui Cao, Yuki Arase, Jacki O'Neill, Zhicheng Dou, Xing Xie, Xiaoyuan Yi, Furu Wei
arXiv Computation and Language
Oct 16

LLMs Infer Cultural Context but Fail to Apply It When Responding

The paper introduces CAPRI, a dataset of conversations designed to test whether large language models (LLMs) can adapt their responses to users’ cultural backgrounds. Experiments show that while LLMs can infer a user’s cultural context and recall relevant conventions, they frequently fail to apply this knowledge when generating answers unless explicitly prompted to do so. The study also finds that LLMs’ cultural adaptations improve as more cues are provided, yet their default priors are not culture‑neutral and sometimes reflect the model’s country of origin.

By Yisong Miao, Jian Zhu, Vered Shwartz
arXiv AI
Oct 5

CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

CuBEs introduces culturally‑situated behavioral evaluations for large language models, adding cultural context to test scenarios and assessments. The authors built a human‑labeled dataset covering 12 cultures, revealing significant cross‑cultural differences that one‑size‑fits‑all judgments miss. Evaluating 13 LLMs shows that culturally situated tests uncover varied behaviors, such as Western political bias versus non‑Western religious or colonial biases, which standard evaluations overlook.

By Hoda Ayad, Tanu Mitra, Abhishek Mukherji
arXiv AI
Jun 4

Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks

arXiv:2601. 22396v2 Announce Type: replace-cross Abstract: Despite the growing utility of Large Language Models (LLMs) for simulating human behavior, the extent to which these synthetic personas accurately reflect world and moral value systems across different cultural conditionings remains uncertain.

By Candida M. Greco, Lucio La Cava, Andrea Tagarelli
arXiv Computation and Language
Oct 27

Naturalistic measure of social norms alignment

arXiv:2605.23420v2 Announce Type: replace Abstract: Social norms reflect shared expectations on acceptable behavior. Measuring social norms alignment remains challenging, with existing approaches typ...

By Yevhen Kostiuk, Kenneth Enevoldsen, Peter Bjerregaard Vahlstrup, M\'arton Kardos, Kristoffer Nielbo
arXiv AI
Jun 9

Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs

arXiv:2606. 08483v1 Announce Type: new Abstract: Background: Consumer-facing large language models are now a common source of health information, and they interpret and personalize responses rather than retrieve them.

By Rahul Gorijavolu, Kaushik Madapati, Pritika Vig, Rawan Abulibdeh, Nikhil Jaiswal, Mahri Kadyrova, Zeamanuel Hailu Tesfaye, Charles Senteio, Paula Maurutto, Leo Anthony Celi