AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,733 stories · RSS feed

arXiv Computer Vision
4d ago

ART of PIV: Agentic Real-Time Particle Image Velocimetry for Active Flow Control

The paper introduces ART of PIV, an automated system that selects the optimal particle image velocimetry pipeline configuration for each image pair based on a chosen measurement objective and speed budget. By dynamically choosing among cross‑correlation, classical optical‑flow, and learning‑based estimators, ART of PIV achieves 12.2‑22.4 % lower average end‑point error compared to the best single estimator at matched speed, and improves closed‑loop control performance in both simulated and physical experiments. The authors demonstrate the method on a simulated velocity regulation task and a tabletop water channel with a fluidic pinball, showing significant gains in accuracy and control effectiveness.

By Antonio Terpin, Francesco Banelli, Alan Bonomi, Raffaello D'Andrea
arXiv Computer Vision
4d ago

REA: A Routed Embodied Agent Runtime for Asynchronous Brain-Cerebellum Control

The paper introduces REA, a unified runtime that integrates a deliberative foundation model (System 2), an embodied router (System 1), and fast reactive policies (System 0) for embodied agents. REA avoids blocking fast execution by using a route memory, a judge gate, and a shared‑budget escalation ladder, enabling asynchronous coordination between the brain and reflexes. Experiments on LIBERO‑Pro, RoboTwin 2.0, BEHAVIOR‑1K, and a real Unitree G1 humanoid show that REA achieves higher success rates and reduces control steps and time compared to single‑policy baselines.

By Pengfei Zhou, Yixing Ma, Zhengfeiyang Zhang, Haoyuan Chen, Zhenglin Wan, Hexin Wang, Wangbo Zhao, Yang You
arXiv AI
4d ago

Structure-Aware Skill Retrieval and Arbitration for Optimization Modeling

The paper introduces “ArchOR”, a skill agent that enhances “OptSkills” with two retrieval modules: Centroid-Aware Representation Learning (CARL) and Signature-Guided Retrieval (S3R). CARL refines cached skill representations using a low-rank prototype residual, while S3R attaches structural signatures to skills, prioritizes candidates via weighted Jaccard overlap, and limits semantic re-ranking to a top‑K set. Using the same DeepSeek-V4.1-Flash backbone and released skill library, “ArchOR” achieves a 65.00% Micro‑Avg. Pass@1 across five benchmarks, outperforming “OptSkills” by 1.64 pp Micro‑Avg. and 2.68 pp Macro‑Avg.

By Anran Liu, Benzhao Hou, Zongsheng Cao, Daichao Wang, Lang Chen, Shitong Chen
arXiv Computer Vision
4d ago

SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement

SketchDirector is a new framework for generating complex scene sketches guided by text prompts. It uses a large language model to break a scene into draft assets described by Bézier curves, then an agentic reasoning loop plans their spatial arrangement with evolving memory, and finally a diffusion-based refinement module improves sketch quality while keeping geometry intact. The authors also release the MCSketch8K dataset, providing editable vector assets, compositional layouts, and object-level manipulation annotations to support future research.

By Yiqi Liang, Xiaodong Cun
arXiv Machine Learning
4d ago

ORION: An Outcome-Driven LLM Advisor for RL-Based Multi-Domain 6G Slice Placement

The paper introduces ORION, a hierarchical orchestrator that uses two LLM agents to translate service intents into structured requests and generate abstract placement plans for 6G network slices across multiple administrative domains. A cooperative multi‑agent RL coordinator then selects cross‑domain partitions, treating the plan as a persistent soft prior, while per‑domain agents place the functions. ORION’s key contribution is an outcome‑driven episodic memory that records network‑condition profiles, proposed plans, and verified feasibility outcomes, feeding this information back to the planner to improve admission rates by 4.6–11.5 percentage points over a planner without memory.

By Mahmoud S. Mayaleh, Fetia Bannour, Nour-El-Houda Yellas
arXiv Machine Learning
4d ago

When the Past Shapes Action: Understanding Embodied Memory

The paper "When the Past Shapes Action: Understanding Embodied Memory" surveys how embodied AI agents can preserve and use past observations, actions, and experiences to improve decision-making in dynamic environments. It categorizes existing approaches along six memory requirements—world-state persistence, task progress, belief maintenance, experiential learning, etc.—and eight architectural styles such as recurrent states, fast weights, and spatial memory. The authors evaluate how current benchmarks test memory-dependent behavior and highlight open challenges in selecting relevant experiences, maintaining consistent beliefs, and consolidating knowledge across episodes.

By Siqi Li, Ziyi Bai, Xinrun Xu, Tinglei Huang, B\"orje F. Karlsson
arXiv Machine Learning
4d ago

Latent-Lens: Visual Perception in Small Language Models Through Latent Communication

Latent‑Lens is a lightweight codec that enables a frozen vision‑language model (VLM) to communicate visual information to a frozen small language model (SLM) directly in latent space, avoiding the lossy bottleneck of discrete text. The codec uses a domain encoder to project VLM hidden states into the SLM’s embedding space and an attention decoder (a LoRA) that teaches the SLM to attend to these visual prefixes, training only about 1.6 M parameters (0.2 % of the combined model size). On visual question answering tasks (CLEVR, GQA) and image captioning (Flickr8k), Latent‑Lens significantly outperforms text‑based communication with the same LoRA budget, achieving up to +69.8 pp improvement on VQA and +18.8 pp on captioning.

By Mo Shahdloo, Faegheh Sardari, Mathew Salvaris
arXiv Machine Learning
4d ago

Controllable User Simulation

The paper formalizes controllable user simulation as a causal inference problem, showing that standard supervised fine‑tuning on post‑hoc trajectory labels introduces a look‑ahead bias that breaks causal consistency. This bias leads to a phenomenon called controllability collapse, where evaluation metric variance explodes under policy shift. The authors propose theoretical conditions for accurate simulation and practical training mitigations—such as a priori controls, step‑wise dynamic controls, and direct policy‑conditioned learning—to eliminate look‑ahead bias, preserve natural variance, and achieve robust zero‑shot generalization to unseen agent behaviors.

By Guy Tennenholtz, Ofer Meshi, Amir Globerson, Uri Shalit, Jihwan Jeong, Craig Boutilier
arXiv Machine Learning
4d ago

AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving

AMMA is a multi‑chiplet, memory‑centric architecture designed to reduce latency in large‑context attention for language models. It replaces GPU compute dies with HBM‑PNM cubes, doubling memory bandwidth and employing a logic‑die microarchitecture, two‑level hybrid parallelism, and a reordered collective flow to minimize communication overhead. Design‑space exploration and evaluations show AMMA achieves 15.5× lower attention latency and 6.9× lower energy consumption compared with the NVIDIA H100.

By Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding
arXiv Machine Learning
4d ago

Mechanistic Interpretability for End-to-End Self-Driving

The paper applies mechanistic interpretability to end‑to‑end autonomous driving by training Sparse Autoencoders on a single‑camera path‑planning model’s internal activations. Unsupervised clustering of the autoencoder’s latent space uncovers interpretable features linked to route intent, vehicle speed, and braking, achieving 98.3% accuracy in recovering route intent without supervision. Visual counterfactual edits confirm that specific latent features correspond to scene changes such as traffic lights and stop signs, and manipulating these latents yields predictable changes in planner outputs; a lightweight adapter trained on these latents reduces trajectory selection error by 3.44% compared to one that ignores them.

By Manav Gagvani, Benjamin Namikas, Sivamurugan Velmurugan, Nysa Kumar, Shrey Sharma, Thomas Peterson, Pratyush Mathur, Hyunseo Chang, Rohan Paleja
arXiv AI
4d ago

Rome Wasn't Built in One Turn: Replay-Grounded Reward Signals for Long-Horizon Freeciv Agents

The paper introduces CivHarness, a system that records game states in Freeciv and replays them to generate long‑horizon reward signals, addressing the problem of sparse deterministic feedback in strategy games. Using CivHarness, the authors train CivTelescope, an LLM judge that predicts which of two positions will lead to a better outcome, outperforming standard language models on scoreboard‑trap pairs. They then employ CivTelescope as a dense reward in a Freeciv agent, CivMarshmallow, achieving significantly higher training gains and outperforming other reward schemes across the full game.

By Yanchen Liu, Xinyu Che, Brihi Joshi, Renzo Fukuda, Junhua Liu, Ruifeng Li, Yunfei Ge, Hang Yan, Xiang Ren
arXiv Machine Learning
4d ago

Do Language Models Know What Compelling Scientific Evidence Looks Like?

The paper introduces EDB-Bio, a benchmark of 96,000 episodes drawn from real biological studies—drug‑perturbation transcriptomics, CRISPR screens, and GWAS—to test whether language models can weigh empirical evidence and predict hypothesis truth. Initial results show that both open‑weight and frontier models are systematically underconfident, matching a trivial base‑rate forecaster in Brier score, but post‑training with proper scoring rules improves calibration and discrimination. In a downstream experiment‑selection task, a calibrated model (Ibis0) outperforms the best frontier model by 2.5×, and Bayes‑factor analysis reveals how models update beliefs, align with expert relevance judgments, and expose biases.

By Conor Igoe, Gabriel Grand, Zane Koch, Albert Bou, Eli Moss, Michaela M. Hinks, Samuel G. Rodriques, Andrew White, Siddharth Narayanan
arXiv Machine Learning
4d ago

TalentRL: Strategy-Guided Reinforcement Learning for Hierarchical Skill Routing

TalentRL is a strategy‑guided reinforcement learning framework that routes natural‑language skills hierarchically, using coarse‑grained talents to guide fine‑grained skill retrieval. The method trains talent routing via reward contrasts and optimizes execution with group‑relative advantages, achieving significant performance gains on open‑domain QA benchmarks and agentic tasks such as ALFWorld and WebShop.

By Huancheng Chen, Xiaodi Sun, Zhaoqiong Huang, Michelle Chang, Jingwen Lu
arXiv Machine Learning
4d ago

FOCUS: First-Order Control Using Optimal Transport Supervision for Imitation from Observations

FOCUS is a new imitation‑learning method that learns from state‑only demonstrations without temporal alignment. It uses local optimal‑transport alignment to match short agent rollouts with sampled expert segments, creating differentiable imitation rewards. A learned critic then bootstraps value beyond the rollout horizon, improving sample efficiency and training stability on contact‑rich locomotion tasks.

By Jo\~ao A. C\^andido Ramos, Lionel Blond\'e, Hugues Vinzant, Jacopo Castellini, Alexandros Kalousis
arXiv Machine Learning
4d ago

Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping

Li$^{2}$SAO is a lightweight variant of Single-rollout Asynchronous Optimization (SAO) that eliminates the need for a pretrained value model by using group-relative actor‑critic bootstrapping. The method first trains the actor with a group-relative advantage and the critic with a group-centered return before switching to single‑rollout Generalized Advantage Estimation. Experiments on single‑turn reasoning and multi‑turn agentic tasks show that Li$^{2}$SAO matches or surpasses the performance of GRPO, especially in early training stages.

By Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao
arXiv Machine Learning
4d ago

Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

Mage‑VL‑Cowork is a 4‑billion‑parameter hybrid computer‑use agent that integrates visual GUI interaction with code and terminal execution in a single policy. It is trained via a progressive five‑stage curriculum—mid‑training, supervised fine‑tuning, and reinforcement learning—starting from a multimodally aligned checkpoint. The training pipeline, Mage‑CUA‑Forge, automates task generation and skill conditioning, producing nearly 300,000 desktop and web tasks while cutting per‑task costs by 82.6%, and the resulting model achieves 66.0% on OSWorld‑Verified, setting a new state‑of‑the‑art for its size.

By Xiaoyi Zhang, Haoqing Wang, Xinjie Zhang, Xiang Ming, Xing Wang, Senqiao Yang, Kaichen Zhang, Zhaoyang Jia, Andong Tan, Xinjiang Wang, Jinghao Guo, Yifei Shen, Xiao Li, Peng Zhang, Xun Guo, Jiahao Li, Shicheng Zheng, Jinglu Wang, Zongyu Guo, Wenxuan Xie, Bin Li, Yan Lu
arXiv Machine Learning
4d ago

In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents

The paper introduces Agentic Soft-Token Compression (ASTC), a training‑free method that uses soft tokens within an LLM agent to compress long‑horizon context. ASTC leverages the model’s recurrent dynamics and Gated Delta Network attention to identify absorbed information, forwarding it as soft tokens while keeping hard‑token values intact. Experiments show ASTC matches or surpasses uncompressed agents, reducing attention pairs by up to 50.68% and outperforming trained compressors by 15.4 to 103.7 times in attention usage.

By Xuning He, Kai Fan, Runyu Zhang, Binghuan Wang, Yuxiang Zhang, Xue Jiang, Yihong Dong
arXiv Computer Vision
4d ago

Coupled or Independent? Fault Coupling in Agentic Medical Vision Systems

The paper investigates how vision‑language agents that expose intermediate outputs can be affected when the sensor reading is unreliable, causing both the tool and reasoning channels to share a visual fault. Using a dermatology agent on the PAD‑UFES‑20 dataset, the authors introduce a fault‑coupling taxonomy distinguishing single‑channel errors from joint errors that agree or disagree, and show that typographic injection leads to detectable directional retrieval errors while physical corruption does not. They find that crop‑first retrieval yields a higher rate of joint wrong agreement (55%) compared to full‑image retrieval (35%), and that hard‑label channel comparison struggles to detect joint failures, limiting disagreement recall to 0.29 for crop‑first and 0.57 for full‑image retrieval, with similar effects observed on the HIBA test set.

By Nishi Kanta Paul, Md Shihabul Islam Shovo, Md Mostafijur Rahman, Jamil Fayyad
arXiv Machine Learning
4d ago

A prospective randomized study of patient-centered AI for clinical visit preparation

A randomized study evaluated a large language model–based chatbot designed to help patients prepare for clinical visits. In the Google Health app, 802 adults used the agent (standard or concise) or a control chatbot, and the agent outperformed the control in clinician‑rated preparation, alignment with patient context, and patient‑reported preparedness, confidence, and satisfaction. The results demonstrate that structured AI‑guided preparation can safely improve patient readiness for visits.

By Xincheng Tan, Matthew Thompson, Xiang Ji, Angelica Willis, Will Vaughan, Roma Ruparel, Michael A. Sanchez, Sean Chang, Joseph Xu, Bhavna Daryani, Preeti Singh, Pinal Bavishi, Kejia Chen, Fan Zhang, Priya Gupta, Ernest Lim, Christina Chen, Bob Lou, Dimitrios Antos, Zach Wasson, Ray Luo, Fadi Yousif, Florence Thng, Mark Malhotra, Joseph Breda, Daniel McDuff, Kavi Goel, Katherine Chou, Dale R. Webster, Avinatan Hassidim, Yossi Matias, Po-Hsuan Cameron Chen, Yun Liu, Mike Schaekermann, Quang Duong, Jackie M. Barr
arXiv Machine Learning
4d ago

TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents

The paper introduces TSBPO, a tree‑structured sampling‑to‑update framework for training interactive language agents. It tackles three intertwined challenges—sampling task and state continuations, assigning delayed feedback, and aggregating token losses—by using a likelihood‑fitted Root/Branch allocator, trajectory and observation‑group advantages, and a clipped token surrogate aggregation. Experiments with a Qwen2.5‑1.5B actor show high success rates on ALFWorld and WebShop benchmarks, demonstrating the effectiveness of the proposed training configuration.

By Rui Yu, Chong Mu, Jinchuan Zhang