Hugging Face Trending Papers

AUDITPLAN: Commit, Then Answer for Auditable Safety Alignment

arXiv AI
Sep 18

AUDITPLAN: Commit, Then Answer for Auditable Safety Alignment

AUDITPLAN introduces a plan-then-answer method for safety alignment in language models, where the model first generates a compact structured safety plan before responding. The plan includes a threat label, intended action, and explicit constraints, allowing machine‑checkable auditing while remaining hidden from end users. Training combines supervised fine‑tuning with reinforcement learning using the FAITHGATE reward, which only rewards correct plans, thereby reducing unsafe shortcuts and improving robustness across Qwen model variants.

By Sai Sri Pushpa Jampani, Kshitij Mishra, Asif Ekbal
arXiv Machine Learning
Aug 27

Training Alignment Auditors via Reinforcement Learning

The paper presents a reinforcement learning approach to enhance large language model (LLM) auditors for alignment tasks. By training policies that investigate target models for hidden behaviors and using an LLM judge to compare investigations, the method improves audit realism and reduces false positives. Experiments show better performance on adversarially fine‑tuned targets and a low false‑positive rate below 1%.

By Paul Rosu, Rowan Wang
arXiv AI
Oct 16

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

PROBE is an algorithm that trains language‑model auditors to probe deployed tool‑using agents for policy violations through realistic, multi‑turn, tool‑mediated dialogue. By framing auditing as a two‑player Markov game and using a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity, PROBE outperforms zero‑shot adversaries in uncovering broader and higher‑yield policy violations. The study compares GRPO, DPO, and GFlowNet optimizations, demonstrating that multi‑turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing existing capability and safety benchmarks.

By Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou
arXiv Machine Learning
Oct 16

CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

The paper introduces Continuous Adversarial Training for Consistent Honesty (CATCH), a method that trains large language models to produce honest responses while preserving their overall capabilities. Using only a few hundred self‑reporting scenarios, CATCH is applied to Llama‑3.3‑70B‑Instruct and Qwen3‑3.6‑27B, and the resulting models show improved honesty across multiple benchmarks, including MASK, Among Us, and AuditBench. Compared to prompting, supervised fine‑tuning, and state‑of‑the‑art honesty steering, CATCH achieves superior honesty while maintaining utility through a novel choice of utility data.

By Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato
arXiv AI
Oct 1

Alignment via Training Against Probes Without Losing Monitorability

The paper proposes probe-guided fine-tuning, a method that uses probes detecting undesired properties in model activations as a direct training signal. Experiments show that continuously updated probes reduce harmfulness and improve honesty while preserving utility, outperforming DPO and inference-time steering in safety-utility trade-offs and robustness to jailbreak and abliteration attacks. Importantly, the concepts remain linearly encoded after fine-tuning, maintaining monitorability.

By Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko
arXiv AI
Sep 25

Auditability Is Not One Property: Rule Overlap, Behavioural Agreement, and Composition in Reinforcement Learning

The paper introduces a protocol for auditing and composing reinforcement‑learning policies using discrete behavioral rules, defining auditability through six testable predicates such as trace integrity and rule coverage. Experiments show that overlapping rule sets do not guarantee behavioral agreement, and that rule‑based fusion often fails to outperform value‑based composition, highlighting limitations in current description layers. The authors provide an evidence‑bounded audit framework and outline future directions for more robust skill composition.

By Liu Hung Ming
arXiv AI
2d ago

AI Alignment via Incentives and Correction

The paper investigates AI alignment by applying law-and-economics models of deterrence and enforcement to agentic AI pipelines. It frames misbehavior as a strategic trade‑off between gains from violations and the likelihood and cost of detection, leading to a fixed‑point problem where stronger penalties can both deter solvers and reduce auditors’ incentive to inspect. The authors formalize this interaction in a two‑agent model, propose a bilevel reward‑design optimization, and demonstrate that adaptive reward profiles improve oversight and reduce hallucinations in an LLM coding pipeline.

By Rohit Agarwal, Joshua Lin, Mark Braverman, Elad Hazan
arXiv AI
Oct 31

Value Probes: Learned Value Functions for Beam-Search Red Teaming

The paper introduces Value Probes, a method that uses linear probing on a language model’s activations to estimate a value function for future rewards in automated red‑team auditing. By treating the audit as a Markov decision process and applying beam search guided by these value estimates, the authors achieve a roughly 50% increase in audit success across 18 target behaviors and three model scales. The approach also demonstrates that Value Probes can be trained on off‑policy data from smaller models, offering a more efficient alternative to costly language‑model follow‑ups.

By Jonathan F. Jao, Andrew Kyle Lampinen