arXiv AI
Aug 11
Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families
arXiv:2608. 08029v1 Announce Type: cross Abstract: Khatri et al.
Scaling Laws for Safety Probes
The paper introduces two scaling laws that predict the performance of safety probes—tools used to monitor language models for harmful behavior—based on model size and the number of labeled examples. By creating six large safety datasets and training probes across multiple model families, the authors show that probe error decreases as a power law with model size and further improves with more examples, with the improvement rate increasing for larger models. These laws enable researchers to forecast probe accuracy and decide whether to invest in more data or a larger model to achieve a desired error threshold.
Related stories
arXiv Machine Learning
Aug 31
The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal Behavior
The paper challenges the assumption that large language models (LLMs) produce deterministic safety responses by examining how random seeds and temperature settings affect refusal decisions. Across four instruction‑tuned models and 876 harmful prompts, 18‑28% of prompts flipped between refusal and compliance depending on sampling configuration, with higher temperatures reducing decision stability. The authors introduce a Safety Stability Index (SSI) and recommend multi‑sample evaluation protocols that account for stochastic variation rather than relying on single‑shot tests.
arXiv Machine Learning
Jul 14
HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models
arXiv:2607. 11475v1 Announce Type: new Abstract: Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance.
arXiv AI
Jun 19
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
arXiv:2606. 20502v1 Announce Type: cross Abstract: Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved.
arXiv AI
4d ago
Linear Probes Transfer from Base to Distilled Language Models despite Rotation
arXiv:2610.35586v1 Announce Type: new Abstract: Linear probes, classifiers that read a language model's internal activations, are widely used to detect harmful requests. They are cheap to train and r...
Hugging Face Trending Papers
Jul 13
HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models
Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task performance, or they use model-agnostic safety classifiers, which may miss failures specific to a given fine-tuned checkpoint.
arXiv AI
Jun 2
DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning
arXiv:2606. 00160v1 Announce Type: cross Abstract: Large language models (LLMs) suffer from degraded safety capabilities even when fine-tuned with benign datasets.
arXiv Machine Learning
Oct 9
Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs
The study investigates how many synthetic conversation samples are required to train activation probes that monitor language models for high‑stakes, harmful, and instruction‑following behaviors. By tracking learning curves across 10–590 samples on various probe models, generators, and prompts, the authors find that the number of samples needed is largely determined by coverage of the monitored concept rather than the difficulty of individual sample types. They show that high‑stakes and harmful probes reach near‑plateau performance with as few as 80 samples, while instruction probes require several times more, and that breadth of sample kinds is more critical for instruction monitoring than for high‑stakes scenarios. The paper also releases evaluation suites, dev sets, and generated data for further research.
arXiv AI
Jun 3
Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability
arXiv:2606. 03648v1 Announce Type: cross Abstract: Adapting foundation large language models to a user's task or preferred style through fine-tuning can result in compromising the model's safety.
arXiv AI
Oct 9
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
The paper introduces Dynamic Boundary Evaluation (DBE), a method that moves beyond static benchmarks by locating the per‑prompt pass probability near 0.5 for large language models. DBE provides a calibrated item bank with difficulty labels, a Skill‑Guided Boundary Search algorithm that works with API‑level access, and an adaptive evaluation protocol that places new models on a unified ability scale. The approach is applied to safety, capability, and truthfulness tasks, expanding coverage without saturation and remaining compatible with existing datasets.
arXiv AI
Sep 2
Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
The paper introduces an action‑class diagnostic framework for multi‑turn tool‑calling in large language model agents, breaking failures into action‑class miscalibration and action‑execution failure across a four‑class action space (TOOL_CALL, ASK, REFUSE, CONFIRM). It defines a self‑revealing upper bound (Acc GAR) to expose state‑grader masking of miscalibration and shows that miscalibration is a significant, previously hidden failure mode, especially for heavily tool‑trained families. The study demonstrates that calibration can be reshaped by context‑only perturbations, but the effects vary widely across models and perturbation mechanisms, underscoring the need for diagnostics beyond aggregate accuracy.
arXiv Machine Learning
Aug 11
Activation Probes Surface Code-Security Signals that the Model's Output Misses
arXiv:2608. 09643v1 Announce Type: cross Abstract: AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated.