← Back to all news
arXiv AI

AgentTrace: Execution-Level Fingerprinting of Black-Box LLM Agents

AgentTrace is an active framework that fingerprints black‑box large language model agents by injecting ambiguous probes into tool interfaces, causing tokenization‑sensitive serialization failures. The initial failures separate model families, while the sequence of repair operations—modeled with lightweight Markov chains—reveals fine‑grained, model‑specific recovery behaviors. This approach achieves 96.0% attribution accuracy even under adversarial identity spoofing and can distinguish intra‑family variants that share identical tokenizers, outperforming unordered behavioral statistics and resisting prompt obfuscation and temperature scaling.

Related stories

arXiv AI
Sep 25

Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior

The paper introduces LIDAR, an active black‑box fingerprinting technique that identifies large language models (LLMs) by observing their decisions and actions during coding‑agent execution. By probing agents on tasks such as post‑edit verification, transient‑failure recovery, and specification–test conflict resolution, LIDAR captures both instance‑level and distribution‑level features of execution trajectories and compares them to clean references using a lightweight probabilistic identifier. Evaluated on 36 models across seven families and two agent harnesses, LIDAR achieves high Top‑1 accuracy and MRR, outperforming four existing fingerprinting and API‑auditing baselines, and demonstrates that agent execution behavior provides model‑identity evidence beyond final outputs.

By Chuyi Wang, Xiaohui Xie, Tongze Wang, Fangchen Luo, Yong Cui
llmsagents
More like this →
arXiv AI
Aug 28

How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation

The paper introduces CTF-ABACUS, a trace-based auditing framework that reconstructs each autonomous language-model agent’s run in Capture-the-Flag (CTF) challenges into evidence‑grounded solve profiles. By decomposing actions into penetration‑testing phases and techniques, it distinguishes genuine exploitation from shortcut methods such as memorized recall or guessing. Applying the framework to 1,435 CTF attempts by six models on 240 challenges shows that only 62‑87% of recovered flags are trace‑verified, highlighting that many successes rely on shallow trajectories rather than true exploitation.

By Kimberly Milner, Minghao Shao, Nanda Rani, Haoran Xi, Venkata Sai Charan Putrevu, Meet Udeshi, Sandeep K. Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Muhammad Shafique, Ramesh Karri
llmsagentsbenchmarks
More like this →
arXiv AI
Jul 16

AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.

By Zichen Ding, Jiaye Ge, Shufan Jiang, Kai Chen, Mo Li, Qingqiu Li, Zehao Li, Zonglin Li, Tiaohao Liang, Shudong Liu, Zerun Ma, Zixing Shang, Wenhui Tian, Zun Wang, Liwei Wu, Zhenyu Wu, Jun Xu, Bowen Yang, Dingbo Yuan, Qi Zhang, Songyang Zhang, Peiheng Zhou, Dongsheng Zhu
llmsagentsbenchmarks
More like this →
arXiv AI
Jun 18

SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents

arXiv:2606. 18356v1 Announce Type: cross Abstract: Tool-using language-model agents introduce security failures that go beyond unsafe text: they can disclose protected objects, write persistent memory, send messages, modify databases, or trigger harmful code and tool effects.

By Yuchuan Tian, Mengyu Zheng, Haocheng Mei, Ye Yuan, Chao Xu, Xinghao Chen, Hanting Chen, Yu Wang
llmsagentsbenchmarkssafety
More like this →