arXiv:2608. 11243v1 Announce Type: new Abstract: We argue that a single structural fact organizes a wide range of phenomena in contemporary AI safety: a semantic safety constraint (e.
arXiv:2603. 10938v2 Announce Type: replace-cross Abstract: Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events.
arXiv:2607. 22929v1 Announce Type: new Abstract: A short fine-tuning run can undo the safety guards of an open-weight model---retraining a refusal-trained assistant to aid weapons development or produce hate speech.
arXiv:2606. 14029v1 Announce Type: new Abstract: Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints.
The paper investigates the difference between cost‑agnostic and cost‑sensitive loss functions when model capacity is limited. It shows that, unlike in ideal infinite‑capacity settings, optimizing a cost‑sensitive objective can yield a strictly better downstream decision than post‑processing a cost‑agnostic model. The authors prove this gap under a hypothesis class that can recover the optimal decision boundary but not the optimal cost‑agnostic hypothesis, and provide a simple example and empirical evidence on UCI datasets with simple models.
arXiv:2609. 10866v1 Announce Type: new Abstract: Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications.
arXiv:2606. 04812v1 Announce Type: cross Abstract: Guaranteeing safety is critical to the deployment of reinforcement learning (RL) agents in the real-world, especially as policies learned using deep RL may demonstrate susceptibility to transition perturbations that result in unknown or unsafe behaviour.
The paper introduces Policy Gradient Penalty (PGP), a single‑loop policy‑space method that enforces convex occupancy‑measure constraints via quadratic‑penalty regularization. PGP constructs pseudo‑rewards to estimate gradients of the penalized objective and uses the classical Policy Gradient Theorem, establishing smoothness and global last‑iterate convergence guarantees for an ε‑optimal constrained entropy value with ε‑bounded constraint violation. The authors validate PGP with ablations on a grid‑world benchmark and demonstrate scalability on two challenging continuous‑control tasks.
The paper investigates how the magnitude of numerical errors in a learning system should be evaluated, arguing that error impact depends on the current learning state, the class affected, and the class weight. By analyzing softmax cross‑entropy, the authors derive exact bounds on loss changes for fixed error sets and use these to provide finite‑error guarantees for losses, probabilities, predictions, and gradients. They then invert these guarantees to certify the maximum allowable primitive error for a given learning‑level requirement, demonstrating through von Mises‑Fisher learning that identical errors can have vastly different effects across states.
arXiv:2607. 16895v1 Announce Type: new Abstract: Safe adaptive control is online adaptation under a safety guarantee on the learning trajectory itself.
The paper proves that using a permissive safety filter in reinforcement learning does not compromise asymptotic performance. By formalizing safety through a safety‑critical Markov decision process and a filtered MDP, the authors show that optimal policies in the filtered MDP achieve the same return as the best safe policy in the original setting. Experiments on Safety Gymnasium confirm zero violations during training and performance that matches or exceeds unfiltered baselines.
How accurate must a numerical approximation be within a learning system? Primitive error alone cannot answer this question: errors of the same magnitude can have very different consequences for losses...