arXiv:2510. 06096v3 Announce Type: replace Abstract: The objectives that Large Language Models (LLMs) implicitly optimize remain dangerously opaque, making trustworthy alignment and auditing a grand challenge.
By Matthieu Bou, Nyal Patel, Arjun Jagota, Satyapriya Krishna, Sonali Parbhoo
Safety tuning pipelines judge only the final answer, which makes it difficult to distinguish robust refusal from two undesirable shortcuts: blanket refusal on benign requests and polished but unfaithf...
AUDITPLAN introduces a plan-then-answer method for safety alignment in language models, where the model first generates a compact structured safety plan before responding. The plan includes a threat label, intended action, and explicit constraints, allowing machine‑checkable auditing while remaining hidden from end users. Training combines supervised fine‑tuning with reinforcement learning using the FAITHGATE reward, which only rewards correct plans, thereby reducing unsafe shortcuts and improving robustness across Qwen model variants.
By Sai Sri Pushpa Jampani, Kshitij Mishra, Asif Ekbal
PROBE is an algorithm that trains language‑model auditors to probe deployed tool‑using agents for policy violations through realistic, multi‑turn, tool‑mediated dialogue. By framing auditing as a two‑player Markov game and using a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity, PROBE outperforms zero‑shot adversaries in uncovering broader and higher‑yield policy violations. The study compares GRPO, DPO, and GFlowNet optimizations, demonstrating that multi‑turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing existing capability and safety benchmarks.
By Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou
The paper proposes probe-guided fine-tuning, a method that uses probes detecting undesired properties in model activations as a direct training signal. Experiments show that continuously updated probes reduce harmfulness and improve honesty while preserving utility, outperforming DPO and inference-time steering in safety-utility trade-offs and robustness to jailbreak and abliteration attacks. Importantly, the concepts remain linearly encoded after fine-tuning, maintaining monitorability.
By Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko
arXiv:2610.16911v1 Announce Type: new
Abstract: Does providing additional context to automated AI auditors improve their effectiveness? Using the Petri tool to audit LLM coding agents, we test the im...
By Ben Slater, Ryan Lundqvist, Leo Richter, Kevin Wei