The paper introduces the concept of Evolutionary Safety for recursive self-improving AI, focusing on how safety properties evolve as an AI system and its successors change. It identifies key risks such as intent drift, error accumulation, and safety-property erosion, and presents a taxonomy covering agent state, model state, evaluation, environment, and update mechanisms. The authors propose methods for discovering and evaluating evolutionary risks, and outline governance principles for modification, selection, authorization, provenance, and recovery, while highlighting open problems for maintaining safety in persistent, adaptive, and recursively self-improving systems.
By Chang Gong, Jingping Bi, Di Yao, Xinjian Liang, Chao Xiang, Ruijie Guo
arXiv:2606. 06114v1 Announce Type: new Abstract: Self-evolving agents improve through continual self-play and self-generated learning signals, but autonomous evolution can also cause capability degradation and safety drift.
By Dianxing Shi, Junqi He, Junhao Chen, Bowen Wang, Yuta Nakashima
arXiv:2609.15915v1 Announce Type: new
Abstract: Meta-reinforcement learning (meta-RL) enables agents to adapt to unseen tasks with limited experience. Despite its promise, the application of meta-RL...
By Zeyang Li, Sunbochen Tang, Navid Azizan
The article surveys safety concerns for self‑evolving agents that continually update their internal state, such as model parameters and memories, from new interactions. It introduces the SAVER framework, which tracks reusable influence, adaptation, violations, exposure, and response to assess whether safety properties persist as agents evolve. The survey finds that legitimate state can become unsafe when its persistence, authority, or scope expands beyond its original conditions, and highlights gaps in current research on descendant repair and longitudinal evaluation.
By Jiahao Chen, Zhou Feng, Oubo Ma, Yichen Yan, Ruixiao Lin, Hangtao Zhang, Linkang Du, Hengyu An, Yong Yang, Jun Liu, Junhao Li, Naen Xu, Chunyi Zhou, Yuan Su, Zehao Jin, Qianli Ma, Leyi Qi, Yiming Wang, Zhe Ma, Yuwen Pu, Mengyao Du, Yuanyi Song, Enhao Huang, Zhihui Fu, Jun Wang, Jinfeng Li, Yuefeng Chen, Hui Xue, Yiming Li, Tianyu Du, Shouling Ji
RePolicy is a reinforcement learning approach designed to invoke safety policies for language model agents by evaluating entire execution trajectories within context-dependent policy libraries. It generates policy-grounded rationales and safety judgments, and is initialized with the PolicyTraj-20K dataset before fine-tuning via GRPO with verifiable rewards and policy-context perturbation. Experiments on six safety benchmarks demonstrate strong safety-detection performance and robust policy invocation across varying contexts.
By Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He
arXiv:2608. 09885v1 Announce Type: new Abstract: The safety of large language model (LLM) agents depends not only on model weights but also on the agent harness that manages context, memory, tools, permissions, and runtime control.
By Wanying Qu, Qinghua Mao, Yu Li, Jiyao Liu, Xin Zhang, Dadi Guo, Yanxu Zhu, Qingyu Liu, Leitao Yuan, Xi Lin, Shanfeng Zhu, Yanwei Fu, Jing Shao, Xia Hu, Dongrui Liu
arXiv:2607. 19292v1 Announce Type: cross Abstract: Current AI safety discourse still focuses disproportionately on visible failures, including obvious harms, dramatic misuse, and hypothetical catastrophic scenarios.
By Gjergji Kasneci, Enkelejda Kasneci
arXiv:2606. 28739v1 Announce Type: new Abstract: Large language models increasingly act as agents: they call tools, move money, delete records, and send messages on a user's behalf.
By Shawn Li, Yue Zhao
arXiv:2608. 11274v1 Announce Type: cross Abstract: The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI.
By Albus W. Ng, Yi Han, Jusheng Zhang, Wenhao Wang
arXiv:2502. 04512v4 Announce Type: replace Abstract: AI advancements have been significantly driven by a combination of foundation models and curiosity-driven learning aimed at increasing capability and adaptability.
By Ivaxi Sheth, Jan Wehner, Sahar Abdelnabi, Ruta Binkyte, Mario Fritz
arXiv:2606. 10228v1 Announce Type: cross Abstract: Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains.
By Kaustubh Mani, Yann Pequignot, Vincent Mai, Liam Paull
SEABench is a benchmark designed to study endogenous misalignment in self‑evolving large language model agents. It contains 48 longitudinal task sequences across various evolution surfaces, task domains, and harm types, and includes an adaptive trajectory discovery pipeline that probes for failures while preserving task intent. Evaluations show that self‑evolution improves task completion rates but often introduces safety failures absent in non‑evolving baselines, with divergent safety behaviors reflected in agents’ chain‑of‑thought reasoning that can be monitored to mitigate unsafe actions.