The paper introduces a method to control reasoning behaviors in Mixture-of-Experts (MoE) language models by labeling sentence-level segments with six distinct reasoning behaviors (planning, execution, restatement, fact, reflection, and conclusion). Linear probes on MoE router logits can predict these behaviors nearly as accurately as probes on the full hidden state, enabling the authors to edit router scores during inference to reduce overthinking. This intervention cuts generated tokens by a median of 30.4% (up to 63.3%) across multiple models and benchmarks without significant loss in accuracy, and often improves it.
By Shengjie Wang, Mohsen Fayyaz, Nanyun Peng
arXiv:2608. 04001v1 Announce Type: cross Abstract: Large language models can solve substantially harder reasoning problems with more inference-time compute.
By Mohsen Hariri, Weicong Chen, Nahal Shahini, Vikash Singh, Kai Ye, Amirhossein Samandar, Debargha Ganguly, Sreehari Sankar, Yanyan Zhang, Shouren Wang, Jerry Peng, Biyao Zhang, Michael Hinczewski, Vipin Chaudhary
arXiv:2607. 28308v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions.
By Huiyuan Tian, Bonan Xu, Shijian Li
Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction.
The paper introduces a two‑level internal readout for mixture‑of‑experts reasoning models. First, it compresses the model’s reasoning states into a 64‑axis semantic frame (J64) that reveals process states not captured by the emitted trace, improving held‑out AUC by 0.096–0.135. Second, it reconstructs this frame from native expert‑routing statistics (R64), achieving high correlation with J64 and preserving most of its predictive gain while enabling low‑overhead, test‑time decision making and improved routing policies.
The paper examines how expert pruning—removing low‑importance experts in Mixture‑of‑Experts models—fails when the router is over‑dispersed, a condition caused by aggressive load‑balancing that spreads tokens nearly uniformly across experts. In this regime, traditional importance signals from router probabilities collapse, making perplexity an unreliable predictor of downstream accuracy; for example, the lowest‑perplexity pruning on gpt‑oss‑20B harms mathematical reasoning while the highest‑perplexity pruning preserves it. To address this, the authors introduce Minimax Expert Score Allocation (MESA), a domain‑aware method that iteratively boosts scores for the most affected domain, achieving minimal worst‑case degradation across domains and outperforming baseline pruning strategies on multiple benchmarks while reducing memory usage.
By Berkcan Kapusuzoglu, Connor Pryor, Sangwoo Cho, Supriyo Chakraborty, Shi-Xiong Zhang, Sambit Sahu, Milind Naphade
The paper introduces the Grounded Integration Measure (GIM), a benchmark of 820 expert‑authored problems designed to test models on tasks that integrate multiple cognitive operations such as constraint satisfaction, state tracking, epistemic vigilance, and audience calibration. GIM emphasizes realistic, broadly accessible knowledge rather than specialized expertise, and uses a judge‑aware 2‑parameter logistic IRT model to produce robust ability estimates across 53 model‑thinking‑level configurations. The authors provide a comprehensive leaderboard of 22 models and 47 test configurations, and conduct an extensive study on how test‑time compute affects model capability, finding that configuration choices like thinking budget and quantization can be as influential as model selection itself.
whyItMatters":"By focusing on integration of multiple cognitive domains, GIM offers a more realistic assessment of model reasoning capabilities than benchmarks that either overemphasize memorization or abstract reasoning alone."
By Rohit Patel, Alexandre Rezende, Steven McClain
The paper introduces a two‑level readout for mixture‑of‑experts reasoning models. First, it compresses the model’s internal reasoning states into a 64‑dimensional semantic frame (J64) that reveals process dynamics beyond the emitted trace. Second, it reconstructs this frame from native expert‑routing statistics (R64), achieving high correlation and preserving most predictive gains while enabling low‑overhead, test‑time decision making.
By Kang Chen, Sihan Zhao, Yixin Cao, Yugang Jiang
arXiv:2606. 26476v1 Announce Type: cross Abstract: Warm-started diffusion samplers accelerate iterative inference, but it is rarely clear which part of the pipeline carries the gain.
By Libo Sun, Po-Wei Harn, Zewei Zhang, Peixiong He, Xiao Qin
The paper introduces Step-wise Truncation Probing (STP) to measure overthinking in reasoning models by generating answers from multiple retained prefixes along a fixed reasoning trajectory. It defines Overthinking Ratio and Accuracy Degradation to quantify the impact of continued reasoning beyond the point of maximum accuracy. Using STP data, the authors construct token‑accuracy Pareto frontiers for various termination strategies and present OTBench, a benchmark that aggregates 70.5 B reasoning tokens across six models and four benchmark groups.
By Weixin Guan, Liang Li, Shiyu Xie, Jiapeng Liu, Bing Li, Peng Fu, Chengyang Fang, Xiaoshuai Hao, Can Ma, Weiping Wang
The paper proposes a unified framework for test‑time reasoning methods, framing them as recursion operators—GROW, PRUNE, and BRANCH—applied to an agent’s reasoning trace. Experiments across five benchmarks and three frontier models show that BRANCH, which samples and selects among multiple reasoning paths, consistently outperforms the other operators and a single‑pass chain‑of‑thought baseline, improving accuracy by an average of 5.98 percentage points. The study also highlights the importance of paired evaluation and careful handling of scoring‑pipeline failures, as these factors can significantly alter comparative outcomes.
By Shengxin Zhang, Xiaomin Wu, Xiyang Wu, Jing Xie
arXiv:2606. 08850v1 Announce Type: cross Abstract: Inference-Time Scaling (ITS) has largely succeeded in verifiable domains like math and coding, where cheap verification enables scalable output selection.
By Giorgio Giannone, Mustafa Eyceoz, Shabana Baig, Shivchander Sudalairaj, Anna C. Doris, Faez Ahmed, Akash Srivastava, Kai Xu