The study investigates whether open‑weight language models can introspect on their own internal computations. Using the Open‑Weight Masked Introspection (OWMI) framework, researchers intervened on various internal components of eight models and asked them to report whether changes had occurred. Across 78,000 measurements, none of the models reliably distinguished real interventions from sham ones, with AUROC values essentially at chance.
Why It Matters: The findings suggest that current open‑weight models lack the ability to audit their own internal states, highlighting a limitation for oversight that relies on a model’s self‑reporting.
By Emilio Ferrara
arXiv:2608.29070v1 Announce Type: new
Abstract: Chain-of-thought (CoT) reasoning traces are increasingly proposed as a mechanism for AI oversight: a monitor inspecting a model's reasoning can, in pri...
By Zimo Shi, Xander Tifft, Wen Xing
A 0.6B language model consistently answers YES to 1,200 logical tests, yet its behavior shows no discrimination. Linear probes reveal the correct verdict with high AUC (0.96) and transfer to unseen structures, but a single scalar readout fails due to a saturated decision threshold offset by +4.6 σ. Adjusting this threshold restores behavior accuracy from 50 % to 81 % and improves higher‑scale models, demonstrating that miscalibrated readouts, not hidden knowledge loss, drive performance gaps.
By Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.
By Yigit Utku Bulut
arXiv:2608. 03291v1 Announce Type: cross Abstract: Chain-of-thought (CoT) reasoning improves large language model (LLM) performance while also providing an observable interface to the model's reasoning process.
By Shashwat Sourav, Aishwarya Balwani
The paper introduces a two‑level readout for mixture‑of‑experts reasoning models. First, it compresses the model’s internal reasoning states into a 64‑dimensional semantic frame (J64) that reveals process dynamics beyond the emitted trace. Second, it reconstructs this frame from native expert‑routing statistics (R64), achieving high correlation and preserving most predictive gains while enabling low‑overhead, test‑time decision making.
By Kang Chen, Sihan Zhao, Yixin Cao, Yugang Jiang
The paper introduces a live trace model that incrementally folds an append‑only event ledger into typed run state, producing per‑consumer views for both human observers and the agent itself. Evaluations show that for observers, the compiled view reduces input tokens by 14–15× and cost by 5–7× while improving accuracy from 0.48 to 0.85–0.87. For agents, maintaining running statistics in per‑step state enables success on 120‑link sequential tasks where full‑context prompting fails, and a prompt‑level scratchpad matches the fold’s accuracy at lower cost.
By Egor Pakhomov, Erik Nijkamp
The paper evaluates safety monitors by measuring recall only on prompts that the target model actually answers, rather than on all harmful prompts. Across several guard systems, recall at a 1% false‑positive rate drops sharply when focusing on answered prompts, with monitors catching refused requests 1.1–6.4 times more often than answered ones. Rewriting prompts to be less explicit dramatically increases compliance and reveals that many harmful requests slip past monitors, especially when phrasing is softened. Fine‑tuning guards on these rewritten prompts improves recall from 0.24 to 0.89 on answered requests and generalizes to unseen benchmarks.
By Sripad Karne
The paper demonstrates that a prompt’s influence is not inherent to the prompt itself but depends on the model, as prompts optimized for one model degrade on another and rankings shift under neutral reformatting. By examining a task‑free structural readout—specifically the fixed‑point behavior of a short‑window argmax map—the authors show that nine tokens of conditioning can move the fixed‑point fraction across most of its range, altering structural classes and model rankings, while instruction tuning has no effect. Attempts to explain this phenomenon through prefix length, content type, bidirectionality, or attention‑sink dominance all fail, indicating that the prompt‑model pair is the fundamental unit of explanation.
whyItMatters":"The study reveals that prompt effectiveness is model‑specific and that simple structural readouts can capture this interaction, challenging assumptions about prompt generality and guiding future prompt‑engineering efforts."
By Nicol\'as Vera Z\'u\~niga
arXiv:2608.29956v1 Announce Type: new
Abstract: Large language models often answer complex reasoning questions without revealing intermediate steps, raising whether they reason latently or complete p...
By Armaan Singh, Ryan Trinh Le, Jasmine Kaur, Abdullah Sultan, Edward Lue Chee Lip, Kiran Nijjer, Adnan Ahmed, Vasu Sharma
arXiv:2608. 03401v1 Announce Type: cross Abstract: Large language models often reason at length before answering, increasing cost and latency.
By Francesca Carlon, Vincent Ginis, Andres Algaba
arXiv:2607. 16451v1 Announce Type: cross Abstract: Chat models sometimes commit to an answer and then produce reasoning that justifies it rather than deriving it -- even when the answer contradicts a task premise.
By Heejin Jo