AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,733 stories · RSS feed

arXiv Computer Vision
Oct 27

DriveUniTok: A Unified Multimodal Tokenizer for Driving World-Action Models

arXiv:2610.35044v1 Announce Type: new Abstract: World models are emerging as a fundamental component of policy learning for real-world agents, enabling agents to predict future world states and plan...

By Jingbo Xu, Hanchao Leng, JiaHui Huang, Hanbing Li, Jingwei Zhao, Kuiyuan Yang, Chen Gao, Guang Chen, Hangjun Ye, Yong Li
arXiv Computer Vision
Oct 27

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

arXiv:2610.33966v1 Announce Type: cross Abstract: People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by mul...

By Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, Raghuraman Krishnamoorthi, Qi Qian
arXiv Computer Vision
Oct 27

Vorch-Omni: Multi-Task Orchestration of Sight and Sound

arXiv:2608.05803v2 Announce Type: replace Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...

By Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wang, Yaohui Wang, Yaole Wang, Yidi Wu, Siqian Yang, Mingyu Yin, Haoran Yu, Gang Yue, Lisai Zhang, Yuting Zhang
arXiv AI
Oct 27

Jointly Modeling Preferences and Imperfect Decision-Making in Large Language Models

The paper introduces the Preference-Behavior Model (PBM), which jointly infers Large Language Model (LLM) preferences and imperfect decision-making mechanisms from observed choices. By applying PBM to three social dilemma games—Public Goods, Collective-Risk, and Common-Pool Resource—the authors show that preferences derived from one dilemma can predict behavior in others, whereas traditional behavioral economic elicitation instruments fail to do so consistently. The study also evaluates how well inferred preferences align with persona instructions, revealing both broad agreement and specific divergences across dimensions, LLMs, and games.

By Axel Abels, Elias Fern\'andez Domingos, Tom Lenaerts
arXiv AI
Oct 27

Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners

The paper "Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners" introduces a benchmark of 324 scenarios across 23 collaboration categories to test how well large language models translate partner evidence into calibrated trust decisions. Each scenario requires three sequential decisions with defined transition dynamics, feedback, and cost schedules, allowing precise analytical evaluation of choices against expected outcomes. The study evaluates seven models and 36 controlled-history probes, finding that while the best frontier model approaches near‑optimal decisions, other models display pathologies such as excessive safeguards, zero‑value verification purchases, prioritizing immediate savings over cooperative learning, and misallocating assistance.

By Mengdi Chu, Jintao Huang
arXiv AI
Oct 27

MultiEvoSkill: Structure-Aware Multi-Agent Skill Evolution for Generalizable Skill Artifacts

MultiEvoSkill is a skill‑optimization framework that transforms skill documents into typed graphs, enabling explicit procedural topology and parallel branches. It employs a heterogeneous agent system to generate revision suggestions based on diverse experiences, and a structure‑aware debate mechanism to resolve edits while respecting dependencies and importance constraints. Experiments show that this approach outperforms state‑of‑the‑art methods and improves cross‑model generalization of skill artifacts.

By Kounianhua Du, Jianxing Liu, Ruoshui Shan, Yucheng Zhang, Ke Li, Xing Sun, Weinan Zhang
arXiv AI
Oct 27

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

The paper investigates how users should set the reasoning‑effort level in large language models, a parameter that trades off inference cost for potential performance gains. Experiments on proprietary and open‑weight models across four benchmarks show that higher effort does not consistently improve accuracy, can be up to 42× more expensive on easy tasks, and alters agent behavior by increasing verification and search steps. The authors propose a simple escalation strategy—starting at the lowest effort and only increasing after a verified failure—that matches or outperforms fixed high‑effort settings while reducing cost.

By Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou, Federico Bianchi
arXiv AI
Oct 27

How Humans Learn Unfamiliar Worlds: A Think-Aloud Dataset on ARC-AGI-3

The paper presents the first dataset that captures how humans learn and use mechanisms in unfamiliar environments, using ARC‑AGI‑3 as a testbed. It includes 144 trajectories from 64 participants across 25 public games, each paired with time‑synchronized actions and think‑aloud audio, and 250 trajectories from large language model agents with and without harnesses. The dataset is annotated for mechanism triggers, action purposes, errors, and recovery, enabling a comparison of human and LLM cognition, highlighting differences in hypothesis testing, exploration, and use of qualitative representations.

By Shuzhi Liu, Archana Warrier, Dat Nguyen, Chengqi Zheng, Kevin Ellis, Yewen Pu
arXiv AI
Oct 27

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

InferPriv is a new benchmark that evaluates inferential privacy leakage in large language model (LLM) assistants. It measures whether an honest-but-curious recipient can deduce a user’s sensitive attributes from an LLM’s reply during routine tasks, a type of leakage not captured by existing direct‑leakage benchmarks. Testing five frontier models, the study finds that inferential leakage rates (16.2–25.2%) far exceed direct leakage, even when direct leakage is near zero under privacy‑aware prompts.

By Pengrun Huang, Yiwei Fu, Luoxi Meng, Kamalika Chaudhuri
arXiv AI
Oct 27

CEPO: A General Framework for Improving Prompt Optimization in Multi-Agent Systems

The paper introduces CEPO, a framework that separates coordination and execution in multi‑agent systems by using a shared coordinator prompt and agent‑specific executor prompts. CEPO guides the optimization of each prompt type according to its role, enabling consistent refinement of coordination rules while allowing execution instructions to remain adjustable. Experiments on 11 benchmarks across four domains show that CEPO consistently improves the performance of four existing prompt optimizers, and additional studies demonstrate better optimization budget efficiency and scaling with more agents.

By Jongha Kim, Seohyun Lee, Jonathan Minho Chung, Jinsung Yoon, Hyunwoo J. Kim
arXiv AI
Oct 27

AutoEnergyDesign: An LLM-Driven Agent for Automatic Energy System Design Optimization

AutoEnergyDesign is an LLM‑driven framework that optimizes building energy system designs by iteratively proposing configuration changes based on prior simulation results and automatically reverting unsuccessful changes via a git‑based mechanism. The agent combines a reasoning model with explicit memory (procedural, semantic, episodic, reflective), a constrained action interface, and a keep/revert signal, and evaluates designs using a 25‑year Net Present Cost model across eight technologies. In a case study with measured building loads, the agent (using Claude Haiku 5.5, Sonnet 5.5, and Opus 5.5) achieved an NPC within 1% of the best baseline after far fewer simulations, outperforming random search, TPE Bayesian optimization, and a $(1+1)$ evolution strategy.

By Amedeo Buonanno, Salvatore Fabozzi, Maria Valenti
arXiv AI
Oct 27

RSIV-Harness: A Recursive Self-Improvement Harness Framework for Open-Domain Conversational Text-to-Visualization

RSIV-Harness is a framework that keeps language model weights fixed while recursively improving the surrounding harness based on failure traces, using a four-layer evaluation to ensure stability. It addresses challenges in open-domain conversational text-to-visualization, such as ambiguous user requests, data source discovery, and iterative chart refinement. The introduced ODAC-Vis benchmark evaluates these capabilities across 640 tasks, and experiments show significant gains in executability and chart structure after a single RSIV-Harness iteration.

By Jiawei Lu, Chen Zhang, Jinwei Lu, Yuanwei Song, Yuanfeng Song, Raymond Chi-Wing Wong
arXiv AI
Oct 27

Cross-Platform Application Migration to HarmonyOS: An Agentic Pipeline with Artifact-Derived Verification

The paper introduces HMOS‑Migrator, an agentic pipeline that converts Android or iOS applications into HarmonyOS NEXT ArkTS projects and validates them by replaying recorded user interactions on a real device. It evaluates migration quality using seven weighted metrics that rely solely on the source application's behavior and generated artifacts, avoiding reference implementations. Across twelve trials on six open‑source apps, the pipeline achieves high structural completeness (83.3–100%) but shows lower on‑device behavior (4.3–63.1%) and UI fidelity (21.1–63.2%) scores, with gradual performance improvements over time.

By Ergun Bi\c{c}ici, Mehmet Kaygusuz, Fatih Turan G\"undo\u{g}du, Emre Filiz, \c{C}a\u{g}atay K{\i}z{\i}lda\u{g}, B\"unyamin Eymen Alag\"oz, Emincan \"Ozcan, Sadi \c{C}elik, Tolunay \"Ozt\"urk, Ege Bulut, Said Aldemir