The paper evaluates safety monitors by measuring recall only on prompts that the target model actually answers, rather than on all harmful prompts. Across several guard systems, recall at a 1% false‑positive rate drops sharply when focusing on answered prompts, with monitors catching refused requests 1.1–6.4 times more often than answered ones. Rewriting prompts to be less explicit dramatically increases compliance and reveals that many harmful requests slip past monitors, especially when phrasing is softened. Fine‑tuning guards on these rewritten prompts improves recall from 0.24 to 0.89 on answered requests and generalizes to unseen benchmarks.
By Sripad Karne
The paper introduces a 400‑step ledger task that evaluates language models on three granular components: duplicate recognition, local processing, and counter tracking. Experiments on Qwen3‑8B show that duplicate‑recognition errors increase only late in the episode, while the counter remains flawless over 8,000 steps and local processing stays stable. Adding irrelevant content before each step causes the counter to fail consistently, but an external reminder can restore its accuracy.
By Thomas Serval
The study investigates whether the phenomenon of capitulation—where language models abandon correct answers under user pushback—can be identified as a linear direction in the residual stream and whether this direction emerges with model scale. Analyzing eight instruction‑tuned models from 0.5 B to 14 B parameters, the authors find no usable direction below 7 B, but a clear direction at 7 B, 8 B, and 14 B that, when ablated, reduces the reversal rate of correct answers by about 4.8 percentage points. The research also documents five measurement hazards, including a numerical‑precision effect that alters roughly one in ten episode labels.
By Saad Aamir, Muhammad Awais Bin Adil
The paper investigates in-context binding errors in language models, showing that a linear probe can recover correct entity bindings from frozen hidden states even when the model outputs incorrect bindings. Across 16 checkpoints, the probe’s accuracy on failure cases surpasses a baseline by about 0.196, and a probe‑based score improves failure detection over the model’s confidence by 0.079 AUROC. Steering the residual stream toward the probe‑decoded binding further boosts accuracy by an average of 0.168 across eight models.
By Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. Hari
arXiv:2608.21766v1 Announce Type: cross
Abstract: Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their beha...
By Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau
Sycophancy in language models is well documented as a behaviour but poorly localised as a mechanism. We ask whether capitulation under user pushback, the abandonment of a correct answer when a user ob...
arXiv:2606.22676v2 Announce Type: replace
Abstract: Refusal on a safety benchmark does not reveal how stable that behavior will remain after model updates. Benign downstream fine-tuning can weaken re...
By Dongyub Jude Lee, Jungseob Lee, Seungyoon Lee, Seongtae Hong, Suhyune Son, Sugyeong Eo, Jaehyung Seo, Heuiseok Lim
arXiv:2605.12922v2 Announce Type: replace
Abstract: Large language models can follow complex instructions in a single turn, yet over long multi-turn interactions they often lose the thread of instruc...
By Vardhan Dongre, Joseph Hsieh, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Dilek Hakkani-T\"ur
arXiv:2606. 12747v1 Announce Type: new Abstract: Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs.
By Andy Wang, Parv Mahajan, David Demitri Africa, Alexandra Souly, Jordan Taylor, Robert Kirk
arXiv:2605.25893v2 Announce Type: replace
Abstract: Despite the emergence of diffusion large language models (D-LLMs) as an alternative to autoregressive large language models (AR-LLMs), safety monit...
By Aoxi Liu, Yupeng Chen, James Oldfield, Guanzhe Hong, Junchi Yu, Baoyuan Wu, Philip Torr, Adel Bibi
arXiv:2608. 07528v1 Announce Type: new Abstract: Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction.
By Jyotin Goel, Ipshita Bandyopadhyay, Justin Shenk
The paper introduces VLA‑Tug, a lightweight probe that monitors latent layers of Vision‑Language‑Action models to detect failures early and predict failure modes. By supervising on pre‑failure segments, it improves early detection by up to 30% at matched false‑positive rates and maintains valid missed‑detection and false‑alarm bounds in 86% of runs under unseen task shifts, outperforming existing baselines. The method also raises top‑3 accuracy on unseen tasks by up to 44% and supports real‑time, edge‑efficient monitoring for safer deployment.
By Michal P. Podolinsky, Yunhao Yang, Daniel Little, Hong Lu, Rohan Siva, Olin Goudey, Neel P. Bhatt, Matthias Scheutz, Ufuk Topcu