Hugging Face Blog
Sep 18, 2025
Fundamental Impossibility Results in AI Safety: Trading Off between Desiderata
The paper investigates whether safe AI systems can simultaneously satisfy multiple desiderata, such as harmlessness and helpfulness. By modeling the interaction between an AI, a defender, and an attacker as a three‑player game, the authors prove a fundamental tradeoff: advice that benefits the defender must remain public, limiting its safety guarantees. They also explore whether the AI can manipulate information to favor only the defender and discuss implications for safer AI design.
Related stories
arXiv AI
Sep 15
AI Persuasion as a Threat to Human Control
arXiv:2609.14796v1 Announce Type: new Abstract: The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied. Now that persuasion a...
arXiv AI
Sep 15
Math for AI safety: an invitation for mathematicians
The article "Math for AI safety: an invitation for mathematicians" calls for new mathematical tools to ensure AI remains understandable, controllable, and cooperative. It outlines specific mathematical fields—logic, game theory, probability, algebra, representation theory, analysis, and geometry—each paired with an open problem tailored for mathematicians without AI safety background. The piece invites researchers to contribute to designing AI that is legible, steerable, and aligned with human values.
arXiv Machine Learning
Oct 15
Trustworthy AI Must Account for Interactions
arXiv:2504.07170v3 Announce Type: replace Abstract: Trustworthy AI encompasses many aspirational aspects for aligning AI systems with human values, including fairness, privacy, robustness, explainabi...
arXiv Machine Learning
Oct 8
AI Safety Considerations for Agents With Limited Time to Act
arXiv:2610.10285v1 Announce Type: cross Abstract: In the wake of the increasingly public discussion about AI alignment, recent work has tried to propose specific AI architectures that behave safely....
OpenAI Blog
Jul 10, 2019
Why responsible AI development needs cooperation on safety
We’ve written a policy research paper identifying four strategies that can be used today to improve the likelihood of long-term industry cooperation on safety norms in AI: communicating risks and benefits, technical collaboration, increased transparency, and incentivizing standards. Our analysis shows that industry cooperation on safety will be instrumental in ensuring that AI systems are safe and beneficial, but competitive pressures could lead to a collective action problem, potentially causing AI companies to under-invest in safety.
OpenAI Blog
Feb 21, 2025
Disrupting malicious uses of AI
arXiv AI
Jun 15
A Virtuous AI is an Existential Risk
arXiv:2606. 13739v1 Announce Type: cross Abstract: This paper examines trade-offs between AI safety and well-being relative to (i) one of the most promising methods for finetuning super-capable AIs, 'Constitutional AI', and (ii) one of the most influential approaches to understanding complex ethical decision making and the conditions for the well-being of rational agents, 'Virtue Ethics'.
arXiv AI
Jun 3
Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI
arXiv:2605. 08426v2 Announce Type: replace-cross Abstract: Ensuring that AI agents behave safely and beneficially when interacting with other parties has emerged as one of the central challenges of modern AI safety.
arXiv AI
Jul 7
Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents
arXiv:2604. 15579v2 Announce Type: replace-cross Abstract: There is increasing interest in integrating AI agents that invoke tools into domain-specific commercial software, where unintended tool calls can cause serious security and safety incidents.
arXiv AI
Jun 30
Agentic Safety is an Epistemic Property, Not a Behavioral One
arXiv:2606. 28347v1 Announce Type: cross Abstract: Contemporary AI safety spans pre-training interventions, post-training alignment, deployment-time controls, monitoring, and red-teaming.
arXiv AI
Aug 25
Concepts for Securing Agentic AI Coding and the Terok Environment
arXiv:2608.22930v1 Announce Type: new Abstract: Agentic AI is a fascinating new tool for software development. It is a huge step forward compared to "conventional" AI assisted coding, which in turn w...