← Back to all news
arXiv AI By Jonathan F. Jao, Andrew Kyle Lampinen

Value Probes: Learned Value Functions for Beam-Search Red Teaming

Read the original on arXiv AI →

The paper introduces Value Probes, a method that uses linear probing on a language model’s activations to estimate a value function for future rewards in automated red‑team auditing. By treating the audit as a Markov decision process and applying beam search guided by these value estimates, the authors achieve a roughly 50% increase in audit success across 18 target behaviors and three model scales. The approach also demonstrates that Value Probes can be trained on off‑policy data from smaller models, offering a more efficient alternative to costly language‑model follow‑ups.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 16

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

PROBE is an algorithm that trains language‑model auditors to probe deployed tool‑using agents for policy violations through realistic, multi‑turn, tool‑mediated dialogue. By framing auditing as a two‑player Markov game and using a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity, PROBE outperforms zero‑shot adversaries in uncovering broader and higher‑yield policy violations. The study compares GRPO, DPO, and GFlowNet optimizations, demonstrating that multi‑turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing existing capability and safety benchmarks.

By Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou
llmsagentsreinforcement-learningbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 27

Training Alignment Auditors via Reinforcement Learning

The paper presents a reinforcement learning approach to enhance large language model (LLM) auditors for alignment tasks. By training policies that investigate target models for hidden behaviors and using an LLM judge to compare investigations, the method improves audit realism and reduces false positives. Experiments show better performance on adversarially fine‑tuned targets and a low false‑positive rate below 1%.

By Paul Rosu, Rowan Wang
llmsreinforcement-learningfine-tuningsafety
More like this →
arXiv AI
Aug 7

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.

By Zhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
llmsagentsbenchmarks
More like this →