arXiv AI

Algorithmic Primitives of Theory of Mind Benchmarks

The paper questions whether Theory of Mind (ToM) benchmarks truly assess theory of mind or merely test parsing and algorithmic state tracking. It introduces PDDL-Mind, a neuro‑symbolic method that parses narratives into the Planning Domain Definition Language (PDDL) and executes candidate actions using predefined update algorithms, yielding program‑verified state traces. Experiments on MMToM, MuMA, and FanToM show that delegating parsing to programs and state tracking to PDDL significantly improves performance, suggesting that current benchmarks mainly penalize unreliable implicit state tracking rather than genuine belief or counterfactual reasoning.

arXiv AI

Aug 25

DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories

DialToM is a Theory of Mind benchmark created from naturalistic human-human dialogues, using a multiple-choice format. It introduces a State-Driven Diagnostic Probe that requires models to predict dialogue trajectories based solely on isolated mental-state profiles, without dialogue context. The evaluation shows that large language models are good at inferring mental states (Literal ToM) but struggle to use them for social forecasting (Functional ToM), while a domain expert scores 100% accuracy, highlighting a clear human‑AI gap.

By Neemesh Yadav, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim

llms benchmarks

arXiv Computation and Language

Oct 23

SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs

SCOPE compiles natural‑language policies into neurosymbolic predicate programs that combine small neural models, deterministic extractors, and logical composition. The system is jointly optimized with adversarially generated counterexamples, achieving higher accuracy than prompt‑optimization, task‑decomposition, and workflow‑generation baselines on real‑world policy benchmarks. Programs trained with the weakest executor transfer to other models, enabling deployment on inexpensive models with stable, interpretable predictions.

By Arnon Mazza, Elad Levi

llms agents fine-tuning benchmarks safety

arXiv AI

Jul 16

Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution

arXiv:2601. 20379v2 Announce Type: replace Abstract: Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption.

By Zhengbo Jiao, Hongyu Xian, Qinglong Wang, Yunpu Ma, Zhebo Wang, Zifan Zhang, Dezhang Kong, Meng Han

llms fine-tuning

arXiv Machine Learning

Aug 19

Recirculation

The paper introduces recirculation, an inference‑time architectural enhancement for foundation models that reduces perplexity and improves accuracy on generation and reasoning tasks without adding significant latency. Recirculation adds a specific form of recurrence, enabling the model to function as a dynamical system that tracks belief states, and is distinct from chain‑of‑thought or depth‑recurrence methods. An adaptive variant requires minimal hyperparameter tuning and achieves notable gains on the Gemma3 family, including a 23% perplexity drop and a 21% accuracy increase on GSM8k.

By Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu

llms

arXiv AI

Oct 21

How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

arXiv:2508.01300v2 Announce Type: replace Abstract: AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Mode...

By Ma'ayan Armony, Albert Mero\~no-Pe\~nuela, Gerard Canal

llms agents nlp benchmarks

arXiv AI

Sep 23

ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures

arXiv:2609.22878v1 Announce Type: new Abstract: Large language model code generation benchmarks primarily evaluate well-resourced languages like Python and Java, where models benefit from abundant tr...

By Aditya Pola, Arkaprava Majumdar, Vineeth N. Balasubramanian

llms benchmarks

arXiv AI

Sep 30

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

arXiv:2609.37233v1 Announce Type: cross Abstract: Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require...

By Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang, Peisen Yao

llms agents benchmarks

arXiv AI

Sep 16

State of Thought Enables Endogenous Reasoning

arXiv:2609.16055v1 Announce Type: cross Abstract: Test-time compute has emerged as a major approach to improving the capabilities of Large Language Models (LLMs). However, existing test-time reasonin...

By Zhiren Gong, Yikun Hou, Zihao Zeng, Ming Xiao, Chau Yuen, Wei Yang Bryan Lim

llms rag multimodal

arXiv Machine Learning

Jun 9

From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning

arXiv:2606. 09092v1 Announce Type: new Abstract: Theory of Mind (ToM) is a must-acquire skill for modern foundation model systems to operate effectively and safely in the real world.

By Jike Zhong, Yuxiang Lai, Ming Li, Yuheng Li, Wuao Liu, Behzad Dariush, Konstantinos Psounis, Shao-Yuan Lo

reinforcement-learning fine-tuning multimodal

arXiv AI

Jun 16

Is Code Better Than Language for Algorithmic Reasoning

arXiv:2606. 15589v1 Announce Type: cross Abstract: For tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism.

By Terry Tong, Yu Feng, Surbhi Goel, Dan Roth

llms benchmarks

arXiv AI

Jul 21

Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering

arXiv:2607. 18100v1 Announce Type: new Abstract: Extended reasoning has become standard for frontier Large Language Models (LLMs), yet the trajectories these models produce remain largely uncontrollable.

By Sheldon Yu, Tong Yu, Xunyi Jiang, Rohan Surana, Gagan Mundada, Sungchul Kim, Lina Yao, Julian McAuley, Junda Wu

llms

Hugging Face Trending Papers

Sep 30

What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling

Recursive reasoning models apply a small shared Transformer block many times to refine a latent state. This gives them large effective depth with few parameters and makes them strong on algorithmic ta...

llms

Related stories