arXiv AI By Yuchuan Fu, Minghe Wang, Xiaohan Yuan, Chenlin Zou, Kaitong Jia, Jinfeng Li, Dongxia Wang

Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks

Read the original on arXiv AI →

The paper introduces RAS‑Eval, a specification‑based security evaluation framework for large language model agents that uses Linear Temporal Logic (LTL) to define utility and security objectives over tool‑call trajectories and dynamic environment observations. Unlike traditional LLM‑as‑a‑Judge or Agent‑as‑a‑Judge approaches, RAS‑Eval derives deterministic verdicts via a model checker, reducing evaluator uncertainty and enabling reliable assessment over long‑horizon tasks. The authors also present a multi‑agent dataset pipeline and a benchmark of 200 normal and 800 attack tasks covering indirect prompt injection, tool poisoning, skill code injection, and tool code injection. whyItMatters":"RAS‑Eval addresses key limitations of existing security evaluations—such as evaluator hallucination and missed transient violations—by providing deterministic, environment‑aware verdicts that are crucial for safely deploying LLM agents in unattended, long‑horizon scenarios."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 17

Symbolic Temporal Supervision of LLM Agents Using Contracts

ContrAgent is a contract‑based framework that provides symbolic temporal supervision for large language model agents. It records an agent’s tool‑call sequence as a trace of checkable predicates and formalizes desired behaviors with assume‑guarantee contracts expressed in linear temporal logic over finite traces (LTLf). Each contract is compiled into a deterministic finite automaton that both gates actions online and evaluates recorded traces offline, enabling deterministic, reproducible verdicts and significantly lower per‑call latency compared to existing LLM‑judge and rule‑based guardrail baselines.

By Yifeng Xiao, Pierluigi Nuzzo