arXiv AI

APEXA: Execution-Integrity Enforcement for Multi-Agent LLM Automation of Synchrotron Data Reduction

arXiv AI
Jul 14

AgentAbstain: Do LLM Agents Know When Not to Act?

arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.

By Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran
arXiv AI
6d ago

Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks

The paper introduces RAS‑Eval, a specification‑based security evaluation framework for large language model agents that uses Linear Temporal Logic (LTL) to define utility and security objectives over tool‑call trajectories and dynamic environment observations. Unlike traditional LLM‑as‑a‑Judge or Agent‑as‑a‑Judge approaches, RAS‑Eval derives deterministic verdicts via a model checker, reducing evaluator uncertainty and enabling reliable assessment over long‑horizon tasks. The authors also present a multi‑agent dataset pipeline and a benchmark of 200 normal and 800 attack tasks covering indirect prompt injection, tool poisoning, skill code injection, and tool code injection. whyItMatters":"RAS‑Eval addresses key limitations of existing security evaluations—such as evaluator hallucination and missed transient violations—by providing deterministic, environment‑aware verdicts that are crucial for safely deploying LLM agents in unattended, long‑horizon scenarios."

By Yuchuan Fu, Minghe Wang, Xiaohan Yuan, Chenlin Zou, Kaitong Jia, Jinfeng Li, Dongxia Wang
arXiv AI
Oct 31

Soei: Structured Observability of Evidence and Intent for Multi-Agent AI Software Engineering

The paper introduces Soei, a structured observability framework for AI coding agents, addressing the need to track what work was done, by whom, and whether it is finished. It evaluates an existing tool, Agenttrail, revealing inconsistencies and security gaps, and presents empirical findings from 45 real coding-agent sessions, highlighting issues such as unverified code edits and credential exposure. Soei comprises an event schema, hash‑chained ledger, PROV‑aligned graph, and verification layer, achieving efficient storage, fast verification, and tamper detection on the same dataset.

By Mohammed Nayeem Khan, K S Sarun Kumar
arXiv AI
Jun 29

LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks

arXiv:2604. 13072v2 Announce Type: replace-cross Abstract: OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments.

By Xiang Long, Li Du, Yilong Xu, RongJian Xu, Qiyanhui Lu, Ying Gao, Qinhua Xie, Fangcheng Liu, Ning Ding, Haoqing Wang, Ziheng Li, Changjiang Zhou, Jianyuan Guo, Yehui Tang
arXiv AI
Oct 15

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...

By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang
arXiv AI
Nov 6

The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks

The paper introduces agent‑native primitives—reusable building blocks that embed a language model within a resource to apply it contextually—and HEELO, a harness‑engineering framework that composes these primitives into task‑specific graphs, verifies outcomes, and iteratively refines or adds primitives. Using eight primitive families, the authors demonstrate significant performance gains across multiple benchmarks, raising Terminal‑Bench 4.0 resolution from 37.3% to 55.5% and improving pass rates on ToolBench and GAIA‑Text‑103. When integrated with HEELO, Claude Opus 5.5 and GPT‑6.1 Sol achieve higher completion counts, faster runtimes, and lower costs on 46 real‑world tasks.

By Haibo Jin, Peng Kuang, Xiaopeng Yuan, Lucheng Fu, Jerry Wang, Xucheng Yu, Yaoning Yu, Dehao Wu, Runguo Li, Colin Su, Jingxuan Sun, Xuyang Liu, Haojing Luo, Haohan Wang
arXiv AI
Aug 28

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

BekchiAI introduces a benchmark and platform for evaluating large language model agents. The benchmark comprises 13 tool‑using ReAct agents across seven task categories, totaling 2,057 deterministic test tasks with verifier‑checkable gold answers. The platform offers web‑based observability, token and latency telemetry, and remote run termination for live agents.

By Mesut Toruk