AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,743 stories · RSS feed

arXiv AI
Sep 24

Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents

Omni-Decision is an omni-modal agent that uses evidence-ledger planning to manage noisy multimodal observations. It replaces the growing dialogue history with a compact evidence ledger that tracks missing, confirmed, and conflicting evidence, allowing the planner to operate on a streamlined context. The system achieves state‑of‑the‑art accuracy on OmniGAIA and WorldSense while operating at a fraction of the cost of larger models.

By Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Yuhao Wang, Junhan Shi, Lingrui Mei, Tianming Yang, Steven Hoi
arXiv AI
Sep 24

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The paper introduces the concept of an agent’s "taste"—its ability to make effective long‑horizon decisions—and presents Taste‑Bench, a new benchmark that automatically generates decision‑fork questions from agent trajectories. Taste‑Bench evaluates models on choosing the best path without seeing future outcomes, revealing that top models answer only about 60% of questions correctly and that later‑appearing evidence makes forks harder. The authors also demonstrate that training a student model to mimic a teacher’s judgment improves decision quality and overall success on held‑out software engineering tasks.

By Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
arXiv AI
Sep 24

How a shared state is described determines whether AI agents synchronize

The article investigates how the textual description of a shared state influences the collective behavior of language‑model agents. By testing 507,112 responses across different model families on a circular coordination task, the authors show that varying the state description (e.g., numerical summaries vs. histograms) can alter whether agents align, split, or fail to coordinate. The study demonstrates that the way a shared state is described is an integral part of the interaction rule that determines collective order.

By Takahiro Ezaki, Naoto Imura, Katsuhiro Nishinari
arXiv Machine Learning
Sep 24

Learning from Failures: Heterogeneous Graph Memory for Small Language Model Tool-Using Agents

The paper introduces FRESH, a Failure-aware Retrieval framework that uses Experience-Structured Heterogeneous graphs to transform past successes and failures into structured external memory for tool‑using agents. By explicitly modeling dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH enables frozen language models to reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on τ‑Bench and AppWorld with multiple open‑source models demonstrate that FRESH consistently improves task success and tool‑use reliability compared to no‑memory agents and other memory‑based baselines.

By Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
arXiv AI
Sep 24

WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents

WAInjectBench introduces the first comprehensive benchmark for detecting prompt injection attacks against web agents, offering a fine‑grained categorization of threats and datasets that include malicious and benign text and image samples. The study systematically evaluates both text‑based and image‑based detection methods across multiple scenarios, revealing that detectors perform well on attacks with explicit instructions or visible perturbations but struggle with subtle or instruction‑free attacks. The authors release the datasets and code to facilitate further research in this area.

By Yinuo Liu, Xilong Wang, Ruohan Xu, Yuqi Jia, Neil Zhenqiang Gong
arXiv Machine Learning
Sep 24

PEARL: A Lightweight Prompt-based Feature Interpreter Framework for Real-Time, Anonymous, and Heterogeneous Collaborative Perception

PEARL is a lightweight, prompt‑embedding framework designed for real‑time, anonymous, and heterogeneous collaborative perception. It uses two parallel, low‑rank visual prompt interpreters—sparse‑detection (LWSD) and dense, domain‑invariant (LWDDI)—to align features and select the appropriate interpreter for newly joining agents without needing their configurations. Experiments on simulated and real datasets show that PEARL improves average precision by 8.2% over random selection, runs in 1.67 ms, and reduces communication cost by up to 34.7× while outperforming state‑of‑the‑art offline methods by 5.6% AP.

By Armin Maleki, Hayder Radha
arXiv Machine Learning
Sep 24

ChronosAttack: Adversarial Tool Scheduling Attacks on LLM Agents

ChronosAttack is a delay‑only scheduling attack that manipulates the timing of authentic tool responses in large language model agents, without altering the content of those responses. By introducing bounded delays, the attack can reorder identical evidence and shift the agents’ final decisions. Experiments on GPT‑5.6 Sol, Gemini 3.6 Flash, DeepSeek V4 Flash, and Claude Sonnet 4.6 show that the attack can cause strong targeted shifts in some models, large opposite shifts in others, and that a single scheduling inversion can dramatically change outcomes. Defenses based on synchronization and order‑consistency reduce the attacker’s influence over observation order.

By Arash Vashagh
arXiv Machine Learning
Sep 24

Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning

The paper introduces Hierarchical Reinforcement and Collective Learning (HRCL), a framework that combines multi‑agent reinforcement learning (MARL) with decentralized coordination. HRCL uses MARL at a high level to generate strategic guidance that limits the decision space for low‑level agents, enabling efficient short‑term coordination while considering long‑term effects. Experiments on synthetic, energy‑management, and drone‑swarm scenarios demonstrate faster convergence and significant reductions in system‑wide and individual costs compared to standalone MARL.

By Chuhao Qin, Evangelos Pournaras
arXiv Computation and Language
Sep 24

Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents

The paper introduces WebMRE, an offline benchmark comprising 541 tasks and 5,293 steps extracted from WebArena trajectories, designed to provide deterministic scoring for web agents without live environments. It enables the first systematic study of how guide sentences and grounded actions reinforce each other, showing that jointly decoding a guide improves element selection accuracy and that the guide acts as a causal instruction channel. The authors fine‑tune models that outperform leading zero‑shot baselines on all offline metrics.

By Chengguang Gan, Yunhao Liang, QingHao Zhang, Shiwen Ni
arXiv AI
Sep 24

Reinforcement Learning with Decomposed Subtasks

The paper introduces Reinforcement Learning with Decomposed Subtasks (RLDS), a method that splits trajectory rewards into per‑subtask shares before policy updates, replacing the scalar advantage used in Group Relative Policy Optimization (GRPO). RLDS employs Subtask‑Decomposed Advantage Estimation (SDAE) to compute group‑relative advantages and distribute credit to tokens based on subtask importance, focusing on steps where a reflection marks a subtask as consequential. Experiments on four benchmarks—FrozenLake, HotpotQA, ScienceWorld, and DeepResearch—show that RLDS improves performance on high‑heterogeneity tasks (ScienceWorld and FrozenLake) and is more compute‑efficient than scalar GRPO for long rollouts.

By Mattie Terzolo, Mikolaj Sacha, Ayan Sinha, Andrew Rabinovich
arXiv AI
Sep 24

PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

PASTABench introduces a benchmark of 1,139 multi-turn trajectories to evaluate proactive safety monitoring in large language models. It formalizes three dimensions of intervention—whether, when, and what risk—to address gaps in step-level isolation and post-hoc trajectory assessment. The study finds that proactive intervention is largely unsolved, with the best model achieving only 40.74% optimal-timing interventions, and reveals that smaller models’ safety scores are often driven by lexical overfitting rather than true risk comprehension.

By Jiapeng Sun, Yujin Zhou, Han Zhu, Pengcheng Wen, Jiayi Zhou, Sirui Han, Yike Guo
arXiv AI
Sep 24

Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools

The paper introduces FWBench, a benchmark for assessing how language models choose and employ time‑series forecasts under budget constraints. Using 1,251 electricity and cycle‑hire cases, the study evaluates agents that select models, histories, and horizons to submit capacities that minimize a loss‑cost objective. Experiments with both hosted and local configurations—including small language models and TSFMs—show that GPT‑6 Astra can achieve superior performance by selectively buying inexpensive short‑horizon forecasts, using only 2.5% of the budget.

By Shunya Nagashima
arXiv AI
Sep 24

DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies

DreamAvoid introduces a test‑time dreaming framework for Vision‑Language‑Action models to anticipate and avoid failures during critical manipulation phases. It uses a Dream Trigger to detect critical phases, samples candidate action chunks via an Action Proposer, and evaluates short‑horizon futures with a Dream Evaluator trained on success, failure, and boundary data. Experiments on real‑world and simulated tasks show DreamAvoid improves task success rates, achieving 72.5% success versus 48.8% for the base policy and 54.4% for GPC‑RANK.

By Xianzhe Fan, Yuxiang Lu, Shenyuan Gao, Xiaoyang Wu, Ruihua Han, Manling Li, Hengshuang Zhao
arXiv AI
Sep 24

Toward Measuring Structural Drift in LLM Communication Loops

The paper introduces a new way to detect drift in stateful language‑model pipelines by treating the sequence of prompt, response, and next prompt as a single unit of analysis. It defines two metrics—communication closure and normalized conditional action contribution—to quantify how well a response aligns with the subsequent prompt and how much it resolves the next reply. Experiments on over 2,200 dialogues show that swapping a response drastically reduces measured contribution, indicating that drift can be detected without labels or predefined rules.

By Wael Hafez, Amir Nazeri, Chenan Wei
arXiv AI
Sep 24

Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memory

The paper introduces ShadowMem, a defensive framework that protects large language model agents from long-horizon threats by maintaining a dedicated safety-focused memory. Inspired by the shadow stack concept, ShadowMem stores safety-critical context throughout an agent’s execution and uses this shadow memory to evaluate the risk of upcoming actions before they are carried out. Experiments show that ShadowMem outperforms existing defenses in detection accuracy, detects most attacks early, and adds minimal overhead to agent performance.

By Yuhui Wang, Tanqiu Jiang, Jiacheng Liang, Charles Fleming, Ting Wang