arXiv AI
Jun 16
Safe Exploration via Policy Priors
arXiv:2601. 19612v3 Announce Type: replace-cross Abstract: Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.
Conformal Policy Control
arXiv:2603. 02196v3 Announce Type: replace Abstract: An agent must try new behaviors to explore and improve.
Related stories
arXiv AI
Jun 10
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
arXiv:2606. 10228v1 Announce Type: cross Abstract: Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains.
arXiv Machine Learning
Jun 2
Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inference
arXiv:2606. 00320v1 Announce Type: new Abstract: We present an online, distribution-free framework for controlling the Conditional Value-at-Risk (CVaR), extending conformal tail risk control to non-stationary and adversarial environments.
arXiv AI
Jun 4
Scenario Generation for Risk-Aware Reinforcement Learning with Probably Approximately Safe Guarantees
arXiv:2606. 04812v1 Announce Type: cross Abstract: Guaranteeing safety is critical to the deployment of reinforcement learning (RL) agents in the real-world, especially as policies learned using deep RL may demonstrate susceptibility to transition perturbations that result in unknown or unsafe behaviour.
arXiv Machine Learning
Jun 15
Utility-Constrained Policy Optimization
arXiv:2606. 14029v1 Announce Type: new Abstract: Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints.
arXiv Machine Learning
Jun 11
Calibrating Decision Robustness via Inverse Conformal Risk Control
arXiv:2510. 07750v3 Announce Type: replace-cross Abstract: Robust optimization safeguards decisions against uncertainty by optimizing against worst-case scenarios, yet their effectiveness hinges on a prespecified robustness level that is often chosen ad hoc, leading to either insufficient protection or overly conservative and costly solutions.
arXiv AI
Jun 16
Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
arXiv:2606. 15385v1 Announce Type: new Abstract: Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety.
arXiv Machine Learning
Jun 5
Conformal Risk-Averse Decision Making with Action Conditional Guarantee
arXiv:2606. 05551v1 Announce Type: cross Abstract: Reliable decision making pipelines powered by machine learning models require uncertainty quantification (UQ) methods that come with explicit safety guarantees.
arXiv AI
Jun 15
Rethinking the Trust Region in LLM Reinforcement Learning
arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
arXiv AI
Jun 2
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning
arXiv:2606. 01991v1 Announce Type: new Abstract: As Large Language Model (LLM) agents increasingly leverage the Model Context Protocol (MCP) to operate in complex environments, the expansion of their action spaces offers agents unsafe capabilities and underscores the risk of power-seeking.
arXiv AI
Jul 14
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
arXiv:2607. 10481v1 Announce Type: cross Abstract: Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile.
arXiv Machine Learning
Jul 27
Safe In-Context Reinforcement Learning
arXiv:2509. 25582v4 Announce Type: replace Abstract: In-context reinforcement learning (ICRL) is an emerging RL paradigm where an agent, after pretraining, can adapt to out-of-distribution test tasks without any parameter updates, instead relying on an expanding context of interaction history.