The paper investigates whether hidden-state probes can identify the type of reasoning failure and the specific step that causes it in large language models. By comparing probes trained on final-answer correctness with those trained using Monte Carlo rollouts, the authors find that standard probes miss a large proportion of recoverable failures and that probe performance depends heavily on training design. Across 4B and 8B models, no probe outperforms a simple rule based on trace length for locating the failing step.
By Shubham Subhnil, Yannis Belkhiter, Erik Miehling, Adam Dahlgren Lindstr\"om, Elizabeth M. Daly
The paper demonstrates that a prompt’s influence is not inherent to the prompt itself but depends on the model, as prompts optimized for one model degrade on another and rankings shift under neutral reformatting. By examining a task‑free structural readout—specifically the fixed‑point behavior of a short‑window argmax map—the authors show that nine tokens of conditioning can move the fixed‑point fraction across most of its range, altering structural classes and model rankings, while instruction tuning has no effect. Attempts to explain this phenomenon through prefix length, content type, bidirectionality, or attention‑sink dominance all fail, indicating that the prompt‑model pair is the fundamental unit of explanation.
whyItMatters":"The study reveals that prompt effectiveness is model‑specific and that simple structural readouts can capture this interaction, challenging assumptions about prompt generality and guiding future prompt‑engineering efforts."
By Nicol\'as Vera Z\'u\~niga
arXiv:2608. 02089v1 Announce Type: new Abstract: Large language models often show users a final response and a short reasoning summary while the full reasoning trace stays hidden.
By Andres Algaba, Francesca Carlon, Lynn Delcon, Marthe Ballon, Bert Verbruggen, Vincent Ginis
arXiv:2607. 11598v1 Announce Type: new Abstract: There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one.
By Bojie Li, Noah Shi
There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one. Both share a hidden limit: they are internal.
arXiv:2608.29956v1 Announce Type: new
Abstract: Large language models often answer complex reasoning questions without revealing intermediate steps, raising whether they reason latently or complete p...
By Armaan Singh, Ryan Trinh Le, Jasmine Kaur, Abdullah Sultan, Edward Lue Chee Lip, Kiran Nijjer, Adnan Ahmed, Vasu Sharma