← Back to all news
arXiv Machine Learning By Ankush Checkervarty

Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs

Read the original on arXiv Machine Learning →

The study investigates how many synthetic conversation samples are required to train activation probes that monitor language models for high‑stakes, harmful, and instruction‑following behaviors. By tracking learning curves across 10–590 samples on various probe models, generators, and prompts, the authors find that the number of samples needed is largely determined by coverage of the monitored concept rather than the difficulty of individual sample types. They show that high‑stakes and harmful probes reach near‑plateau performance with as few as 80 samples, while instruction probes require several times more, and that breadth of sample kinds is more critical for instruction monitoring than for high‑stakes scenarios. The paper also releases evaluation suites, dev sets, and generated data for further research.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 5

Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild

The paper investigates whether adding a short classification instruction after a user’s prompt improves the ability of activation probes to detect malicious inputs in large language models. Across 13 safety benchmarks and three open‑weight model families, a classification suffix consistently boosts out‑of‑distribution detection (up to ~4 AUC points) compared to no suffix, and the benefit transfers to multi‑position pooling probes used in production. The improvement stems from the classification format itself rather than the specific content of the instruction, though the optimal suffix varies with the model and readout type.

By Elad David, Max Fomin
llmsagentsbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 23

Scaling Laws for Safety Probes

The paper introduces two scaling laws that predict the performance of safety probes—tools used to monitor language models for harmful behavior—based on model size and the number of labeled examples. By creating six large safety datasets and training probes across multiple model families, the authors show that probe error decreases as a power law with model size and further improves with more examples, with the improvement rate increasing for larger models. These laws enable researchers to forecast probe accuracy and decide whether to invest in more data or a larger model to achieve a desired error threshold.

By Yuchen Wu (University of Washington), Jerry Wei (Anthropic)
llmssafety
More like this →