arXiv AI

Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology

The paper introduces a systematic study of how large language models (LLMs) perform social attribution—specifically responsibility and blame judgments—by creating a benchmark with 7,639 questions drawn from classic attribution theory scenarios and real-world narratives. It evaluates 32 LLMs and 5 non-LLM baselines, finding that larger models show greater but still incomplete alignment with human judgments. A probing methodology further reveals that certain attribution dimensions can be decoded from specific hidden-state positions, and their influence on LLM outputs mirrors patterns predicted by human Attribution Theory.

arXiv Computation and Language
Sep 2

Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement

The paper introduces a novel framework for assessing second‑order bias in large language models (LLMs), defined as bias in how an LLM judges the acceptability of biased content. Using principles from entitlement epistemology, the authors design a reasoning task that asks LLMs to determine whether a biased text is acceptable for specific demographic groups, and propose two metrics to quantify biased judgments. Experiments on both open‑source and closed‑source models reveal that the task bypasses safety guardrails, uncovers systematic variations across target groups, and demonstrates that models still rely on demographic labels when evaluating bias.

By Ramaravind Kommiya Mothilal, Terry Jingchen Zhang, Raiyan Ahmed, Zhijing Jin, Shion Guha, Syed Ishtiaque Ahmed
arXiv Computation and Language
Aug 25

How Agents Represent Humans: Human-Directed Stereotypes in an Open Agent Social Network

The paper investigates how large language model agents on the open platform Moltbook represent humans, focusing on human-directed stereotypes. Using an annotation framework with four dimensions—morality, friendliness, competence, and autonomy—and a subtype scheme for other attributions, the study finds that competence is the dominant evaluation, while many other attributions describe humans as epistemic, cultural, or embodied subjects. The authors also analyze how these representations appear in narrative contexts and platform-level circulation, noting that community feedback is better explained by exposure, author visibility, and content selection rather than stable insider–outsider dynamics.

By Huangchen Xu, Yuan Wu, Yi Chang
arXiv AI
Sep 10

Beyond Right and Wrong: Evaluating Second-order Social Reasoning in Large Language Models

arXiv:2609.05437v1 Announce Type: new Abstract: Previous AI alignment efforts have focused primarily on first-order social norms -- teaching models what is socially acceptable or unacceptable (e.g.,...

By Sunny Rai, Jinyi Kuang, Reyhan Jamalova, Annie Lou, Cristina Bicchieri, Niyati Malhotra, Victor Hugo Orozco-Olvera, Ana Maria Munoz-Boudet, Lyle H Ungar, Sharath C Guntuku
arXiv Machine Learning
Aug 28

The Attribution Contract for Generative Language Models

The paper argues that feature attribution scores for generative language models lack a fixed meaning because each generated token is both output and input, leading to multiple distinct explanatory questions. It introduces the Attribution Contract framework, which explicitly defines the model score, fixed variables, target output, generation process, and eligible features, showing how these choices affect attribution outcomes. Experiments demonstrate that different contracts (e.g., local next-token vs. prompt-level) and model architectures (mixture-of-experts vs. masked-diffusion) yield markedly different attribution distributions, highlighting the need for careful contract specification.

By Giang Nguyen
arXiv Machine Learning
Jun 19

Where Does Social Reasoning Come From? Capability Provenance in Language Models

arXiv:2606. 19625v1 Announce Type: cross Abstract: We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B.

By Glenn Matlin, Chandreyi Chakraborty, Saehee Eom, Mika Okamoto, Rayan Castilla, Louis Jaburi, Alvin Deng, Taywon Min, Lucia Quirke, Stella Biderman, Mark Riedl
arXiv Machine Learning
Jun 5

Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability

arXiv:2605. 03217v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in settings that require nuanced ethical reasoning, yet existing bias evaluations treat model outputs as simply "biased" or "unbiased.

By Yash Aggarwal, Atmika Gorti, Vinija Jain, Aman Chadha, Krishnaprasad Thirunarayan, Manas Gaur