arXiv AI
Aug 10
Social World Models
arXiv:2509. 00559v3 Announce Type: replace Abstract: Humans intuitively navigate social interactions by simulating unspoken dynamics and reasoning about others' perspectives, even with limited information.
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
SocialMaze is a new benchmark designed to evaluate large language models on social reasoning tasks that involve deep reasoning, dynamic interaction, and information uncertainty. It comprises six tasks drawn from social deduction games, everyday interactions, and digital communities, and includes automated checks and human validation to ensure data quality. Experiments with twelve LLMs reveal that stronger chain‑of‑thought reasoning improves performance on deeper inference tasks, while uncertainty consistently hurts results; targeted fine‑tuning on curated reasoning traces can markedly enhance structured social‑reasoning abilities.
Related stories
arXiv Computation and Language
Aug 24
ZenGen: Social Mind for LLMs
ZenGen is a comprehensive framework designed to develop socially intelligent large language models (LLMs). It introduces SoMBench, a psychology‑grounded benchmark with 3 primary and 17 secondary dimensions, and shows that current LLMs perform poorly on social cognition tasks. ZenGen combines supervised fine‑tuning, on‑policy distillation, and rubric‑based reinforcement learning to improve internalization, while Actio provides deployment‑time grounding with typed supports such as PRISM, Starling, SAGE, and gated RAG, yielding significant performance gains across benchmarks.
arXiv Machine Learning
4d ago
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
The paper investigates whether reasoning representations—explanations for large language model outputs—aid humans in evaluating those outputs. A controlled human study tested six reasoning formats across tasks of varying complexity, measuring structural understanding, error detection, and trust calibration. Results revealed a mismatch: participants favored planning- and decomposition-based representations, yet simpler chain-of-thought traces better supported verification, trust, and interpretability, while preferred formats increased calibration risks.
arXiv Computation and Language
Sep 1
SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos
SocialReasonBench is a new video‑multiple‑choice QA benchmark designed to test socially grounded reasoning in interactive narrative videos. It uses branching gameplay footage from *Detroit: Become Human*, where player choices create alternative social outcomes that can be verified against the game’s script and flowchart. The benchmark includes seven reasoning dimensions—such as intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent—and employs a multi‑agent pipeline to curate clips, ground answer labels, and generate theory‑guided questions with diagnostic distractors.
arXiv Machine Learning
Jun 4
Reasoning Shift: How Context Silently Shortens LLM Reasoning
arXiv:2604. 01161v2 Announce Type: replace Abstract: Large language models (LLMs) exhibiting test-time scaling behavior, such as extended reasoning traces and self-verification, have demonstrated remarkable performance on complex, long-term reasoning tasks.
arXiv AI
Jul 8
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.
arXiv AI
Aug 11
LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routing
arXiv:2608. 08889v1 Announce Type: new Abstract: Recommendation systems thrive on personalization, where ''correctness'' is rarely a binary truth but a matter of subjective human preference.
arXiv AI
Jun 10
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
arXiv:2606. 11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems.
arXiv AI
Aug 3
M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities
arXiv:2601. 02854v2 Announce Type: replace Abstract: As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning.
arXiv AI
Aug 25
A Modular Multitask Reasoning Framework Integrating Spatio-temporal Models and LLMs
The paper introduces STReason, a modular multitask reasoning framework that combines large language models with spatio‑temporal models to handle complex natural language queries without task‑specific fine‑tuning. STReason decomposes queries into interpretable programs, executes them to produce numerical results and detailed, computation‑grounded explanations, thereby reducing hallucinations. The authors evaluate the system on a new benchmark and show it outperforms advanced LLM baselines, with human studies confirming its credibility and practical utility.
arXiv AI
Aug 24
Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning
The paper introduces a method to improve test-time scaling (TTS) for large language models by using multi-agent systems (MAS) to split long reasoning chains into manageable contexts. A new dataset, M500, containing 500 multi-agent collaborative reasoning traces, is used to fine‑tune open‑source models, enabling them to learn collaborative patterns and outperform their base versions. An adaptive scaling strategy with a "CEO" agent is proposed to dynamically guide reasoning depth, and experiments in the AgentVerse framework confirm the effectiveness of the approach.
arXiv AI
Aug 12
From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models
arXiv:2608. 10444v1 Announce Type: cross Abstract: Large language models (LLMs) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains.