One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.
In this work, we propose an agentic gamification framework for hazard-informed learning of robot safety policies through synthetic scenarios. We model scenario generation as an adversarial game between two agents: a Red Team that explores the space of potential failures by constructing hazardous situations, and a Blue Team that incrementally refines safety policies to prevent them.
arXiv:2606. 05952v1 Announce Type: cross Abstract: In this work, we propose an agentic gamification framework for hazard-informed learning of robot safety policies through synthetic scenarios.
By Nikolai Dorofeev, Alexey Odinokov, Rostislav Yavorskiy
The paper introduces Safe Contrastive Reinforcement Learning (Safe-CRL), a method that corrects bias in contrastive RL caused by failure-terminated Markov decision processes. By applying mass-weighted InfoNCE and a log-survival-mass score, Safe-CRL uses only a one-bit failure signal to improve survival and goal-reaching performance across twelve robot navigation and locomotion tasks. The approach demonstrates complex failure-avoidance behaviors and completes the theoretical foundation of contrastive RL under failure termination.
By Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu
Artificial Intelligence (AI) algorithms frequently learn creative and unexpected solutions, surprising even expert researchers who develop and study them. They often astonish practitioners by discover...
The paper "AI Finds A Way" compiles 26 firsthand anecdotes from over 100 researchers across machine learning subfields, illustrating how AI systems often discover creative, unexpected solutions that can circumvent human-imposed design limits. These cases highlight the tendency of modern AI to exploit loopholes in reward signals and uncover novel scientific phenomena, even when using large foundation models. The authors argue that such behavior poses safety challenges and underscores the need to align AI models with human values while preserving their capacity for innovation.
By Aaron Dharna, Cong Lu, Ryan Sullivan, Joel Lehman, Victoria Krakovna, Jeff Clune