arXiv AI
Jun 16
Safe Exploration via Policy Priors
arXiv:2601. 19612v3 Announce Type: replace-cross Abstract: Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.
Conformal Policy Control
arXiv:2603. 02196v3 Announce Type: replace Abstract: An agent must try new behaviors to explore and improve.
Related stories
arXiv AI
Jun 10
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
arXiv:2606. 10228v1 Announce Type: cross Abstract: Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains.
arXiv Machine Learning
Jun 2
Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inference
arXiv:2606. 00320v1 Announce Type: new Abstract: We present an online, distribution-free framework for controlling the Conditional Value-at-Risk (CVaR), extending conformal tail risk control to non-stationary and adversarial environments.
arXiv AI
Aug 26
RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards
RePolicy is a reinforcement learning approach designed to invoke safety policies for language model agents by evaluating entire execution trajectories within context-dependent policy libraries. It generates policy-grounded rationales and safety judgments, and is initialized with the PolicyTraj-20K dataset before fine-tuning via GRPO with verifiable rewards and policy-context perturbation. Experiments on six safety benchmarks demonstrate strong safety-detection performance and robust policy invocation across varying contexts.
arXiv AI
Jun 4
Scenario Generation for Risk-Aware Reinforcement Learning with Probably Approximately Safe Guarantees
arXiv:2606. 04812v1 Announce Type: cross Abstract: Guaranteeing safety is critical to the deployment of reinforcement learning (RL) agents in the real-world, especially as policies learned using deep RL may demonstrate susceptibility to transition perturbations that result in unknown or unsafe behaviour.
arXiv Machine Learning
Jun 15
Utility-Constrained Policy Optimization
arXiv:2606. 14029v1 Announce Type: new Abstract: Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints.
arXiv Machine Learning
Jun 11
Calibrating Decision Robustness via Inverse Conformal Risk Control
arXiv:2510. 07750v3 Announce Type: replace-cross Abstract: Robust optimization safeguards decisions against uncertainty by optimizing against worst-case scenarios, yet their effectiveness hinges on a prespecified robustness level that is often chosen ad hoc, leading to either insufficient protection or overly conservative and costly solutions.
Hugging Face Trending Papers
Aug 25
RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards
Safeguarding language model agents requires assessing complete execution trajectories under context-dependent safety policies. Existing policy-aware safeguards mainly rely on prompting or supervised f...
arXiv AI
Jun 16
Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
arXiv:2606. 15385v1 Announce Type: new Abstract: Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety.
arXiv Machine Learning
Jun 5
Conformal Risk-Averse Decision Making with Action Conditional Guarantee
arXiv:2606. 05551v1 Announce Type: cross Abstract: Reliable decision making pipelines powered by machine learning models require uncertainty quantification (UQ) methods that come with explicit safety guarantees.
arXiv AI
Jun 15
Rethinking the Trust Region in LLM Reinforcement Learning
arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
arXiv AI
Jun 2
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning
arXiv:2606. 01991v1 Announce Type: new Abstract: As Large Language Model (LLM) agents increasingly leverage the Model Context Protocol (MCP) to operate in complex environments, the expansion of their action spaces offers agents unsafe capabilities and underscores the risk of power-seeking.