arXiv AI
Jun 2
Safety Alignment of LMs via Non-cooperative Games
arXiv:2512. 20806v3 Announce Type: replace Abstract: Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment.
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
arXiv:2506. 07468v4 Announce Type: replace Abstract: Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities.
Related stories
arXiv Machine Learning
Jun 5
Alignment Risks from Capability-Seeking RL Training
arXiv:2602. 12124v2 Announce Type: replace Abstract: While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk arises from capability-seeking RL training in vulnerable environments.
arXiv AI
Aug 6
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
arXiv:2608. 04317v1 Announce Type: cross Abstract: Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied.
arXiv AI
Jun 16
Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
arXiv:2606. 15385v1 Announce Type: new Abstract: Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety.
Hugging Face Trending Papers
Aug 5
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied. Meanwhile, recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have improved LLM reasoning, but their integration into cybersecurity remains elusive due to the absence of suitable benchmark environments and interaction datasets.
arXiv Machine Learning
Jul 1
Addressing Over-Refusal in LLMs with Competing Rewards
arXiv:2606. 31748v1 Announce Type: new Abstract: Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones.
arXiv AI
Jun 4
A Systematic Investigation of RL-Jailbreaking in LLMs
arXiv:2605. 07032v2 Announce Type: replace-cross Abstract: The evolution of generative models from next-token predictors to autonomous engines of complex systems necessitates rigorous safety hardening.
arXiv AI
Jun 12
Beyond Runtime Enforcement: Shield Synthesis as Defensibility Analysis for Adversarial Networks
arXiv:2606. 13621v1 Announce Type: new Abstract: Shielded reinforcement learning is typically presented as a runtime safety mechanism that compiles temporal-logic specifications into automata restricting an agent's actions.
arXiv Machine Learning
Jul 24
PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses
arXiv:2603. 13026v2 Announce Type: replace Abstract: Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents.
arXiv AI
Jul 24
Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind
arXiv:2604. 11666v2 Announce Type: replace-cross Abstract: As large language models (LLMs) become the engine behind conversational systems, their ability to reason about the intentions and states of their dialogue partners (i.
Hugging Face Trending Papers
Jun 13
Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment
Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution. Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.
arXiv AI
Jun 19
LLM agent safety, multi-turn red-teaming, jailbreak benchmarks, adversarial robustness, safety-critical systems
arXiv:2606. 20408v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized.