Detection Is Not Early Warning: Evaluating Token-Level Alarms on Language Model Failures
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The paper evaluates safety monitors by measuring recall only on prompts that the target model actually answers, rather than on all harmful prompts. Across several guard systems, recall at a 1% false‑positive rate drops sharply when focusing on answered prompts, with monitors catching refused requests 1.1–6.4 times more often than answered ones. Rewriting prompts to be less explicit dramatically increases compliance and reveals that many harmful requests slip past monitors, especially when phrasing is softened. Fine‑tuning guards on these rewritten prompts improves recall from 0.24 to 0.89 on answered requests and generalizes to unseen benchmarks.
The paper introduces a 400‑step ledger task that evaluates language models on three granular components: duplicate recognition, local processing, and counter tracking. Experiments on Qwen3‑8B show that duplicate‑recognition errors increase only late in the episode, while the counter remains flawless over 8,000 steps and local processing stays stable. Adding irrelevant content before each step causes the counter to fail consistently, but an external reminder can restore its accuracy.
The study investigates whether the phenomenon of capitulation—where language models abandon correct answers under user pushback—can be identified as a linear direction in the residual stream and whether this direction emerges with model scale. Analyzing eight instruction‑tuned models from 0.5 B to 14 B parameters, the authors find no usable direction below 7 B, but a clear direction at 7 B, 8 B, and 14 B that, when ablated, reduces the reversal rate of correct answers by about 4.8 percentage points. The research also documents five measurement hazards, including a numerical‑precision effect that alters roughly one in ten episode labels.
The paper investigates in-context binding errors in language models, showing that a linear probe can recover correct entity bindings from frozen hidden states even when the model outputs incorrect bindings. Across 16 checkpoints, the probe’s accuracy on failure cases surpasses a baseline by about 0.196, and a probe‑based score improves failure detection over the model’s confidence by 0.079 AUROC. Steering the residual stream toward the probe‑decoded binding further boosts accuracy by an average of 0.168 across eight models.
arXiv:2608.21766v1 Announce Type: cross Abstract: Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their beha...
Sycophancy in language models is well documented as a behaviour but poorly localised as a mechanism. We ask whether capitulation under user pushback, the abandonment of a correct answer when a user ob...