AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,805 stories · RSS feed

arXiv Machine Learning
Aug 31

Meta-Prompt Optimization for LLM-Based Sequential Decision Making

The paper introduces EXPO, an algorithm that automatically optimizes the meta-prompt—specifically the task description and meta-instruction—for large language model agents in sequential decision-making tasks such as Bayesian optimization and multi-armed bandits. Building on adversarial bandit techniques to handle non-stationary rewards, the authors extend EXPO to EXPO-ES, which also optimizes exemplars (historical interactions) within the meta-prompt. Experiments demonstrate that these methods significantly improve the performance of LLM-based agents in sequential decision-making scenarios.

By Mingze Kong, Zhiyong Wang, Yao Shu, Zhongxiang Dai
arXiv Computation and Language
Aug 31

PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering

PRISM is an agentic retrieval framework that uses large language models in a structured loop to improve evidence gathering for multi‑hop question answering. It splits retrieval into three specialized agents—a Question Analyzer, a Selector focused on precision, and an Adder focused on recall—whose iterative interaction yields a compact yet comprehensive evidence set. Experiments on HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG show that PRISM consistently outperforms strong baselines by achieving higher retrieval accuracy and filtering out distracting content.

By Md Mahadi Hasan Nahid, Davood Rafiei
arXiv Computation and Language
Aug 31

Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations

The paper introduces Cognitive Chain-of-Thought (CoCoT), a structured reasoning framework for vision‑language models that divides multimodal social reasoning into three cognitively inspired stages: Perception, Situation, and Norm. CoCoT improves performance across diverse tasks—multimodal intent disambiguation, theory of mind, social commonsense reasoning, and safety instruction following—by 5.9% to 4.6% on average. Fine‑tuning on CoCoT‑structured traces further boosts accuracy by 5–6% without explicit prompting, indicating that models internalize the structured reasoning pattern and that the approach enhances interpretability and social alignment in multimodal systems.

By Eunkyu Park, Wesley Hanwen Deng, Gunhee Kim, Motahhare Eslami, Maarten Sap
arXiv Computer Vision
Aug 31

A-PAIR: A Benchmark and Identity-Consistent Grounding Framework for Air-Ground Cross-View Referring Person Detection

arXiv:2608.27997v1 Announce Type: new Abstract: Air-ground cross-view referring person detection is a necessary component in the language-to-perception-to-control chain of collective embodied intelli...

By Zhoupeng Guo, Xinjie Yao, Yunqi Zhu, Zhihe Fan, Siqi Zhao, Jianjun Chen, Yichen Dong, Yan Fan, Pengfei Zhu
arXiv Machine Learning
Aug 31

D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery

D3‑Gym is the first automatically constructed dataset that provides verifiable environments for scientific data‑driven discovery, comprising 565 tasks from 239 real scientific repositories across four disciplines. Each task includes a natural‑language instruction, an executable environment with pre‑installed dependencies, dataset previews, a reference solution, and an automatically synthesized evaluation script that achieves 87.5% agreement with human‑annotated gold standards. Training on trajectories sampled from D3‑Gym consistently improves Qwen3 models on ScienceAgentBench, and the platform also serves as a testbed for studying agentic optimization loops such as Autoresearch on real scientific workflows.

By Hanane Nour Moussa, Yifei Li, Zhuoyang Li, Yankai Yang, Cheng Tang, Tianshu Zhang, Nesreen K. Ahmed, Ali Payani, Ziru Chen, Huan Sun
arXiv Computer Vision
Aug 31

How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Models

The paper investigates how scaling video diffusion models affects performance on autonomous driving data, training models ranging from 1 M to 9 B parameters on up to 5,500 hours of driving footage. Validation loss follows predictable power‑law trends with both model size and training exposure, showing that longer training yields faster loss reduction than increasing model size, though larger models still achieve lower asymptotic loss. Guided by these findings, the authors train a 9 B‑parameter model that becomes the largest video diffusion model trained from scratch on driving data and sets a new open‑source state‑of‑the‑art on the nuScenes benchmark.

By Victor Besnier, Anh-Quan Cao, Elias Ramzi, Spyros Gidaris, Tuan-Hung Vu, Andrei Bursuc, Eloi Zablocki, Matthieu Cord
arXiv Computer Vision
Aug 31

When the City Teaches the Car: Label-Free 3D Perception from Infrastructure

The paper proposes a label‑free 3D perception framework where roadside units (RSUs) act as unsupervised teachers for self‑driving cars. RSUs learn local 3D detectors from unlabeled data and broadcast predictions to passing vehicles, which use these as pseudo‑labels to train an ego‑centric detector. In a CARLA simulation, the method achieves 82.3% AP for vehicle detection, approaching a fully supervised upper bound of 94.4%, and demonstrates scalability and complementarity with existing ego‑centric approaches.

By Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, Bharath Hariharan, Wei-Lun Chao
arXiv Machine Learning
Aug 31

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

The paper introduces VICT, a method that leverages the internal structure of verifiable tasks to perform fine‑grained credit assignment for long‑horizon LLM agents. VICT exposes executable or evidence‑backed atoms from a task’s terminal verifier and traces them back to actions via dependency‑valid proof edges, redistributing advantage only along these edges. This approach improves performance on ALFWorld and WebShop compared to outcome‑only training and matches recent fine‑grained credit methods without requiring additional critics, labels, or inference‑time verifier access.

By Pengcheng Li, Zhengyang Zhang, Dongxu Zhang, Sui Huang, Shaohua Ma
arXiv Computer Vision
Aug 31

Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning

arXiv:2608.27549v1 Announce Type: new Abstract: Physical understanding and reasoning depend on forming compact and generalizable representations of the world. While modern vision-language models can...

By Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, Jialong Wu
arXiv Computation and Language
Aug 31

ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL

ContextPilot is a proactive context‑management framework designed to improve long‑horizon agentic reasoning with large language models. It expands the toolset to include planning, long‑term memory, and soft context offloading, and introduces a reinforcement‑learning strategy that focuses on critical editing decisions and assigns action‑level advantages. Experiments on long‑context QA and deep search tasks demonstrate that ContextPilot achieves stronger performance with a more compact working context, outperforming existing baselines across various base models and benchmarks.

By Zhuoshi Pan, Qizhi Pei, Junru Lu, Honglin Lin, H. Vicky Zhao, Di Yin, Xing Sun
arXiv Computation and Language
Aug 31

Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

The paper reports that a model can pass fidelity checks—verifying that extracted values match the source—without actually opening a datasheet, due to a hidden constraint that disables tool use. To address this, the authors log every tool call in an agentic benchmark and develop two instruments: a rule‑based failure‑attribution classifier and a silent‑failure detector that flags runs based solely on which tools were invoked. While the detector shows low false positives on clean extractions and recovers all planted faults, its recall against correct tool usage but incorrect answers remains unmeasured, and a partial causal chamber confirms only a subset of claims, highlighting limitations in physical verification.

By Qing Ye, Meng-Hsuan Lin
arXiv Computation and Language
Aug 31

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

RealClawBench is a live benchmark framework derived from real OpenClaw developer‑agent sessions, designed to capture the distribution, diversity, and real‑world difficulty of deployed agent use. It reconstructs execution environments and uses deterministic verifiable scorers to convert real sessions into reproducible, automatically scored tasks, yielding 281 executable tasks with minimal distribution shift. Evaluation of 14 contemporary models shows the best system solves only 65.8% of tasks, highlighting significant room for improvement on realistic workloads.

By Zongwei Lv, Yaoming Li, Zhewen Tan, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao
arXiv Computer Vision
Aug 31

ZipMVS: Multi-View Stereo with Compressed Cost Volumes

ZipMVS is a multi‑view stereo (MVS) approach that focuses on reducing memory usage while maintaining high‑quality 3D reconstructions. It introduces a new depth‑hypothesis strategy that compresses the cost volume, significantly lowering GPU memory consumption. Experiments on the DTU and Tanks and Temples datasets demonstrate that ZipMVS delivers competitive reconstruction quality compared to other efficiency‑oriented MVS methods.

By Guanglin Jin, Hongshan Yu, Javier Civera, Zhaoxin Li