The paper introduces an unsupervised post‑training objective that maximizes mutual information between chain‑of‑thought (CoT) explanations and model outputs to enhance CoT monitorability. Experiments on three model organisms show that this method recovers most or all lost monitorability in degraded CoT settings and further improves monitorability, unsafe behavior prediction, and consequential reasoning in undegraded policies. The results argue that monitorability should be treated as a trainable attribute and that a dedicated post‑training stage can protect CoT from direct optimization pressure.
By Usman Anwar, Tim Bakker, Christos Louizos, David Krueger
OwlSight is a new chain‑of‑thought (CoT) monitoring system that uses a customizable Factor strategy to adapt to a wide range of tasks and monitoring needs. By incorporating fine‑grained and coarse‑grained factors, OwlSight can detect specific behaviors such as email leakage or broader privacy concerns across multiple dimensions like capability, safety, and trustworthiness. Experiments on 15,971 reasoning trajectories show that OwlSight outperforms general‑purpose LLMs by 1.32‑30.57 points at fine granularity and 11.71‑27.81 points at coarse granularity, achieving state‑of‑the‑art results on OpenAI’s CoT monitorability benchmark and demonstrating robustness to model changes, factor rephrasings, CoT compression, and obfuscation.
By Lewen Yan, Yanxu Zhu, Jie Zhang, Jing Shao
arXiv:2604. 23488v3 Announce Type: replace Abstract: Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models.
By Lichen Li, Hengguang Zhou, Yijun Liang, Tianyi Zhou, Cho-Jui Hsieh
arXiv:2603. 28590v3 Announce Type: replace Abstract: Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs.
By Han Wang, Yifan Sun, Brian Ko, Mann Talati, Jiawen Gong, Zimeng Li, Naicheng Yu, Xucheng Yu, Wei Shen, Vedant Jolly, Huan Zhang
arXiv:2604. 23488v2 Announce Type: replace Abstract: Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models.
By Lichen Li, Hengguang Zhou, Yijun Liang, Tianyi Zhou, Cho-Jui Hsieh
The paper introduces Gradient Fingerprint (GRIFT), a technique that uses a model’s internal gradient computations to detect reward hacking in reinforcement learning with verifiable rewards. GRIFT compresses gradients of a chain-of-thought (CoT) conditioned on a prompt into a compact representation, which is then used to assess whether the CoT reflects reward hacking. Experiments on math, code, and logical reasoning benchmarks show GRIFT outperforms baselines by over 25% and, when integrated into a rejection fine‑tuning pipeline, reduces reward hacking while improving task performance.
By Songtao Wang, Quang Hieu Pham, Fangcong Yin, Xinpeng Wang, Jocelyn Qiaochu Chen, Greg Durrett, Xi Ye