arXiv AI
Aug 10
Social World Models
arXiv:2509. 00559v3 Announce Type: replace Abstract: Humans intuitively navigate social interactions by simulating unspoken dynamics and reasoning about others' perspectives, even with limited information.
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
SocialMaze is a new benchmark designed to evaluate large language models on social reasoning tasks that involve deep reasoning, dynamic interaction, and information uncertainty. It comprises six tasks drawn from social deduction games, everyday interactions, and digital communities, and includes automated checks and human validation to ensure data quality. Experiments with twelve LLMs reveal that stronger chain‑of‑thought reasoning improves performance on deeper inference tasks, while uncertainty consistently hurts results; targeted fine‑tuning on curated reasoning traces can markedly enhance structured social‑reasoning abilities.
Related stories
arXiv Computation and Language
Aug 24
ZenGen: Social Mind for LLMs
ZenGen is a comprehensive framework designed to develop socially intelligent large language models (LLMs). It introduces SoMBench, a psychology‑grounded benchmark with 3 primary and 17 secondary dimensions, and shows that current LLMs perform poorly on social cognition tasks. ZenGen combines supervised fine‑tuning, on‑policy distillation, and rubric‑based reinforcement learning to improve internalization, while Actio provides deployment‑time grounding with typed supports such as PRISM, Starling, SAGE, and gated RAG, yielding significant performance gains across benchmarks.
arXiv Machine Learning
Sep 10
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
The paper investigates whether reasoning representations—explanations for large language model outputs—aid humans in evaluating those outputs. A controlled human study tested six reasoning formats across tasks of varying complexity, measuring structural understanding, error detection, and trust calibration. Results revealed a mismatch: participants favored planning- and decomposition-based representations, yet simpler chain-of-thought traces better supported verification, trust, and interpretability, while preferred formats increased calibration risks.
arXiv Computation and Language
Oct 16
CLUE: Uncovering Social Context for Pragmatic Understanding
CLUE is a two‑stage training pipeline that teaches models to discover and use social context cues within a conversation for pragmatic understanding. It combines supervised evidence reasoning with a GRPO reward that credits new, evidence‑supported social inferences while penalizing excessive reasoning effort. Experiments show that CLUE improves weighted F1 scores by 17.3% in‑domain and 9.3% out‑of‑domain, reduces reasoning tokens by over 80%, and aligns its explanations more closely with human‑selected conversational evidence.
arXiv Computation and Language
Sep 1
SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos
SocialReasonBench is a new video‑multiple‑choice QA benchmark designed to test socially grounded reasoning in interactive narrative videos. It uses branching gameplay footage from *Detroit: Become Human*, where player choices create alternative social outcomes that can be verified against the game’s script and flowchart. The benchmark includes seven reasoning dimensions—such as intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent—and employs a multi‑agent pipeline to curate clips, ground answer labels, and generate theory‑guided questions with diagnostic distractors.
arXiv Machine Learning
Jun 4
Reasoning Shift: How Context Silently Shortens LLM Reasoning
arXiv:2604. 01161v2 Announce Type: replace Abstract: Large language models (LLMs) exhibiting test-time scaling behavior, such as extended reasoning traces and self-verification, have demonstrated remarkable performance on complex, long-term reasoning tasks.
arXiv AI
Oct 16
AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks
AlphaDiana is a measurement system designed to evaluate agents in verifiable reasoning tasks by providing modular interfaces for datasets, agents, environments, and scorers. It maintains harness-native control loops while separately managing file reuse and memory retention, enabling both macro-level analyses of entire agents and micro-level interventions on components such as tools, skills, and memory. Experiments on open-source models and harnesses across multiple benchmarks reveal that harnesses and component interventions do not consistently improve reasoning accuracy, and that token usage metrics must be interpreted alongside correctness.
arXiv AI
Jul 8
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.
arXiv AI
Aug 11
LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routing
arXiv:2608. 08889v1 Announce Type: new Abstract: Recommendation systems thrive on personalization, where ''correctness'' is rarely a binary truth but a matter of subjective human preference.
arXiv AI
Sep 16
Verifiable Social Reasoning for LLM Assistants
arXiv:2609.17496v1 Announce Type: new Abstract: LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (...
arXiv AI
Sep 17
WordPolo: Evaluating Language Models Through Iterative Semantic Feedback
WordPolo is a word‑finding task that evaluates language models by having them guess an unknown target word and receive semantic similarity feedback. Participants start with no knowledge, make iterative guesses, and receive distance scores that guide them through semantic space. The study tests recent LLMs, LRMs, humans, and a heuristic on 1,500 puzzles, revealing that while solve rates vary widely, many models make meaningful progress and exhibit human‑like strategies, highlighting the importance of assessing reasoning processes, not just final accuracy.
arXiv AI
Jun 10
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
arXiv:2606. 11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems.