REALHOP introduces a behavioral auditing framework to assess multi‑hop reasoning by measuring the Behavioral Necessity Rate (BNR), which quantifies how often removing targeted evidence prevents correct answers. Across five benchmarks, the framework reveals a wide gap between annotated reasoning chains and actual evidence dependence, with panel‑mean BNR ranging from 16.6% to 48.9%. By re‑binding entities, factorizing relations, adding competing paths, and placing evidence at traceable locations, REALHOP raises BNR dramatically—from 27.4% to 94.4% on MuSiQue questions—while maintaining high overall accuracy and improving performance on long‑context tasks.
By Jiawen Tao, Xiaokun Yuan, Yaoming Li, Chenxu Liu, Mengzhou Wu, Tong Yang, Maxm Pan
arXiv:2606. 30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer?
By Wenlong Wang, Fergal Reid
The paper introduces a causal, layerwise audit method called the CoT Mediation Index (CMI) to evaluate whether chain-of-thought (CoT) prompting truly influences a language model’s internal computation. By comparing performance degradation from patching CoT-token hidden states against matched control patches, the authors find that CoT influence is often confined to narrow reasoning windows and can be nearly absent even when the model produces fluent rationales. The study shows that models explicitly tuned for reasoning exhibit stronger mediation, while Mixture-of-Experts models display more distributed mediation, indicating that CoT faithfulness varies across models and tasks.
By Anish Sathyanarayanan, Aditya Nagarsekar, Aarush Rathore
The paper investigates whether latent chain-of-thought (latent CoT) models truly perform reasoning by examining twenty-one model-task configurations across different training recipes, tasks, and backbones. Using chain deletion, content replacement, probes, and trajectory analysis, the authors find that on logical tasks the latent chain often contains an answer before the chain starts and can be deleted with little effect, while on arithmetic tasks the chain’s content is essential. The study introduces a taxonomy that distinguishes what a chain represents from how a model uses it, clarifying that readable latent states alone do not prove reasoning.
By Jingjie Qiu, Sheldon Yu, Xunyi Jiang, Jiajin Liu, Zihan Huang, Zhendong Chu, Tong Yu, Julian McAuley, Junda Wu
The paper introduces a method for analyzing how interventions in a prompt affect a language model’s reasoning by using behavioral counterfactuals. It trains a module to edit the residual stream so that the next-token distributions for a fixed response match between a source and a counterfactual prompt, outperforming standard activation reconstruction. The resulting attribution maps and visualizations help identify causally important locations for counterfactual recovery and guide hypothesis testing with activation patching, as demonstrated in a case study on Gemma-4-E4B-it and DeepSeek-R1-0528-Qwen3-8B.
By Shawn Im, James Oldfield, Adel Bibi, Sharon Li
arXiv:2607. 06648v1 Announce Type: new Abstract: Latent reasoning methods perform multi-step inference entirely in the model's continuous hidden states, promising more compact and efficient reasoning.
By Hengyu Jin, Shu Yang, Di Wang