AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,743 stories · RSS feed

arXiv AI
Sep 24

OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

arXiv:2609.23407v2 Announce Type: replace-cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challengin...

By Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong
arXiv AI
Sep 24

Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity

The paper introduces JAZ, a minimalist LLM agent framework that centers on a single primitive called “invoke”, which allows an LLM to write and execute arbitrary code, including recursive calls, while treating all inputs and interaction history as variables in the code environment. JAZ provides built‑in hooks for constraints and monitoring but relies solely on prompting, without external tools, memory systems, or file‑system access. Experiments show that JAZ “invoke” outperforms specialized external harnesses such as Letta (MemGPT) and ACE on long‑horizon recall tasks and continual self‑improvement, achieving higher accuracy at lower cost.

By Zhening Li, Joshua Liu, Mateja Vukelic, Nicole Shen, Supriya Lall, Amitayush Thakur, Alex Zhang, Omar Khattab, Jonathan Light, Armando Solar-Lezama
arXiv AI
Sep 24

Building Socio-Affective Artificial Intelligence for Interactive Multi-Agent Simulations

The article presents design principles and a software architecture, AGIMUD, for enabling interaction between humans and multiple agents in dynamic simulated worlds. It integrates socially-aware reasoning, emotional agent behavior, a multimodal human interface, and distributed AI processing to support real‑time, multi‑user dungeon (MUD) environments. The work builds on current AI/AGI and transformer‑based conversational agents to create sustainable, governance‑aware human‑agent reasoning systems.

By David Berga
arXiv Computation and Language
Sep 24

UniDataAgent: An Ontology-Grounded Agent for Enterprise Question-to-Report Automation

UniDataAgent (UniDataAgent) is an ontology‑grounded system designed to automate enterprise question‑to‑report tasks while preserving organization‑specific semantics. It separates semantic acquisition from online execution, with an Ontology Acquisition and Validation (OAV) stage that builds versioned ontologies from metadata, business knowledge, and expert input, and a Question‑to‑Report Execution (QRE) stage that retrieves semantic contracts, coordinates skills and data tools, validates results, and produces evidence‑linked reports. In a deployment across 27 enterprise tables and thousands of metric types, ontology construction took a few hours versus a week manually, and report generation took minutes versus several working days, achieving 95.0% strict accuracy on real business questions compared to 72.5% for document RAG.

By Yutai Duan, Yahui Zhao, Zhangti Li, Yu Ma, Zhenfeng Qi, Shaoyang Yuan, Jing Fan, Jie Liu
arXiv AI
Sep 24

Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

The paper audits silent failures in agent-to-tool interactions within the ToolUniverse environment, focusing on 15 scientific tools. It identifies 91 failures—primarily missing data or inconsistent search/filtering—occurring mainly in the API and wrapper layers, and shows how these silent failures can propagate downstream into seemingly valid outputs. The authors propose contextual reliability and recommend testing, disclosure, monitoring, and measurement strategies for the agent-tool pipeline.

By Shreya Gopalan, Devansh Singh, Sundaraparipurnan Narayanan
arXiv AI
Sep 24

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures

The paper introduces a counterfactually anchored evidence attribution approach for multi‑turn large language model safety failures. It presents a new dataset of 1,762 conversations, including adversarial, benign twins, and high‑risk vocabulary variants, and trains a lightweight hierarchical model that accurately predicts safety violations and attributes them to specific user turns and token spans. The model achieves high detection performance (F1 = 0.988) and significantly reduces adversarial confidence when top‑attributed tokens are removed, while maintaining low false‑positive rates on benign conversations.

By Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz Khan
arXiv Computer Vision
Sep 23

ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model

ForeDrive introduces a planning-relevant latent world model that is asymmetrically coupled to a Diffusion Transformer planner. The model learns multi‑horizon latent futures with a JEPA‑style world model, while planning gradients update the shared encoder and stop‑gradient routing trains the predictor with forecasting losses only. Gated visual fusion, future‑status injection, and Trajectory‑Adaptive Bias are used to guide trajectory generation without overriding current observations, achieving high performance on NAVSIM benchmarks using only front‑view images and pure imitation learning.

By Sinuo Wang, Zichong Gu, Yuhan Huang, Wenxin Wen, Xun Yang, Yiqing Zhang, Xingyu Zhang, Ningyu Che, Jie Ling, Qiankun Yu, Wei Liu, Jing Xu, Xinggang Wang
arXiv AI
Sep 23

Divergent strategies and convergent outcomes in autonomous materials discovery

The study examined sixteen independent runs of a single autonomous materials discovery model, each given the same database of 12,499 metal‑organic frameworks and tasked with optimizing methane storage. The agents’ strategies varied widely, screening between 100 and 5,000 structures and constructing 2,253 hypothetical frameworks, yet all converged on the same high‑performance materials frontier near 200 cm³ cm⁻³. Even when half the agents were subjected to stricter checks, the overall conclusions remained unchanged, revealing both robust findings and a shared error stemming from an incomplete database entry.

By Jihan Kim
arXiv AI
Sep 23

Self-Healing Harness for Runtime Oversight of Agent Self-Modification

The paper introduces a self‑healing harness that enforces admission control over language‑model agents’ self‑modifications. The harness runs a Detect‑Notice‑Heal‑Validate loop, allowing agents to propose rule changes that are only granted persistent authority after demonstrating improvement on a failure case without regressing on protected cases. Across 16 benchmark runs, the harness rejected many locally beneficial proposals that caused collateral regressions, while improving task‑completion scores and reliability.

By Sina Tayebati, Divake Kumar, Nastaran Darabi, Ranganath Krishnan, Amit Ranjan Trivedi