arXiv AI

Sapien: A Stateful Policy Engine for Autonomous AI Agents

Sapien is a policy engine that enforces stateful contextual policies for autonomous AI agents, specifying allowed tool‑call sequences with an extended regular expression that includes stateful predicates, deferred policy generation, and scoped semantic checks. The system maintains performance close to an unconstrained agent while significantly reducing malicious actions, ruling out 93‑95% of attacks on AgentDojo and 62‑85% on Toolathlon, outperforming traditional tool allowlists on long‑horizon tasks.

arXiv AI

Sep 23

ActGov: Governing LLM Agent Actions via Policy-Constrained Validation

ActGov is a runtime enforcement framework that validates each action proposed by a large language model (LLM) agent before it interacts with external tools, ensuring that actions stay within task‑scoped authorization boundaries and comply with dynamically constructed policies. It builds policies from tool specifications, benign tasks, and failure traces, verifying updates via SMT‑based counterexample checking. In evaluations on AgentDojo and AgentDyn benchmarks, ActGov consistently reduces indirect prompt‑injection attack success while maintaining task utility, outperforming existing defenses.

By Kaiyuan Zhang, Yuke Peng, Ke Jiang, Yinqian Zhang

llms agents benchmarks

Hugging Face Trending Papers

Sep 21

ActGov: Governing LLM Agent Actions via Policy-Constrained Validation

Large language model (LLM) agents increasingly execute long-horizon workflows through external tools, allowing untrusted outputs to influence subsequent actions and exceed user authorization. Existing...

llms agents benchmarks

arXiv AI

Sep 1

AgenTRIM: Tool Risk Mitigation for Agentic AI

arXiv:2601.12449v2 Announce Type: replace-cross Abstract: AI agents are autonomous systems that combine LLMs with external tools to solve complex tasks. While such tools extend capability, improper t...

By Roy Betser, Amit Giloni, Shamik Bose, Sindhu Padakandla, Chiara Picardi, Lidor Erez, Roman Vainshtein

llms agents benchmarks safety

arXiv AI

Aug 18

Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement

arXiv:2608. 14590v1 Announce Type: new Abstract: LLM agents increasingly perform irreversible real-world actions, including database updates, API calls, file operations, and autonomous use of tools.

By Pierre Dantas, Lucas Cordeiro, Ehsan Nowroozi, Tihanyi Norbert

llms agents safety

arXiv AI

4d ago

Environment Steering: Using Data Flow Control to Improve Agent Utility and Safety

arXiv:2609.35807v1 Announce Type: cross Abstract: LLM agents can make unsafe tool calls even when instructed to behave safely. Existing defenses constrain agents before execution, modify tool inputs/...

By Charlie Summers, Prajwal Raghunath, Aaditya Pai, Mayur Kulkarni, Zhuo Zhang, Oliver Kennedy, Eugene Wu

llms agents safety

arXiv AI

Jul 3

ElephantAgent: Contextual State Continuity in Agentic Systems

arXiv:2607. 01919v1 Announce Type: new Abstract: Agentic systems enhance their capabilities by invoking external tools and maintaining persistent memory.

By Jiankai Jin, Xiangzheng Zhang, Zhao Liu, Wenzhuo Xu, Dongdong Yang, Deyue Zhang, Quanchen Zou

agents safety

arXiv AI

Aug 26

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards

RePolicy is a reinforcement learning approach designed to invoke safety policies for language model agents by evaluating entire execution trajectories within context-dependent policy libraries. It generates policy-grounded rationales and safety judgments, and is initialized with the PolicyTraj-20K dataset before fine-tuning via GRPO with verifiable rewards and policy-context perturbation. Experiments on six safety benchmarks demonstrate strong safety-detection performance and robust policy invocation across varying contexts.

By Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He

llms agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Sep 2

OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleets

arXiv:2609.00015v1 Announce Type: new Abstract: AI agents powered by large language models are evolving from isolated assistants into heterogeneous systems in which multiple agents, planners, control...

By Dongsheng Chen, Xiangyu Zhao, Xin Yao, Xuetao Wei

llms agents benchmarks safety

arXiv AI

Jun 19

Efficient and Sound Probabilistic Verification for AI Agents

arXiv:2606. 20510v1 Announce Type: cross Abstract: Securing AI agents that operate in complex digital environments has become a critical need, and runtime monitoring approaches that formulate and enforce policies expressed in a formal language like Datalog offer a promising solution.

By Alaia Solko-Breslin, Pramod Kaushik Mudrakarta, Mihai Christodorescu, Somesh Jha, Krishnamurthy Dj Dvijotham

agents benchmarks

arXiv AI

Aug 3

Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

arXiv:2607. 29254v1 Announce Type: new Abstract: AI agents extend large language models (LLMs) with external tools, enabling them to perform complex tasks and translate model outputs into consequential real-world actions.

By Minghui Pan, Jiayuxuan Yang, Yuanyuan Yuan, Yu Jiang, Zhenpeng Chen

llms agents benchmarks safety

Hugging Face Trending Papers

Aug 25

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards

Safeguarding language model agents requires assessing complete execution trajectories under context-dependent safety policies. Existing policy-aware safeguards mainly rely on prompting or supervised f...

llms agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 2

SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents

arXiv:2606. 02302v1 Announce Type: cross Abstract: Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services.

By Hao Cheng, Changtao Miao, Tianle Song, Yin Wu, He Liu, Erjia Xiao, Junchi Chen, Xiaoyu Shi, Yichi Wang, Jing Yang, Taowen Wang, Jinhao Duan, Mengshu Sun, Peiyan Dong, Xuan Shen, Yang Cao, Renjing Xu, Kaidi Xu, Jindong Gu, Bo Zhang, Jize Zhang, Chenhao Lin, Philip Torr, Chao Shen

llms agents benchmarks safety

Related stories