AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,805 stories · RSS feed

arXiv AI
Aug 24

A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications

The paper surveys how large multimodal models (LMMs) enhance agentic frameworks that combine perception, memory, reasoning, planning, and action. It examines the integration of multiple modalities—text, images, audio, and video—through delegated, late‑fusion, and early‑fusion architectures, and maps these designs to agent capabilities. The survey also reviews multimodal agentic systems in robotics, web navigation, multimedia content creation, and video understanding, evaluating performance, efficiency, and scalability trade‑offs.

By Neel Mokaria, Rishie Raj, Dheeraj Baiju, Xiaoqian Shen, Shraman Pramanick, Kevin Qinghong Lin, Arda Senocak, Mike Zheng Shou, Philip Torr, Mohamed Elhoseiny, Yapeng Tian, Ruohan Gao, Salman Khan, Sayan Nag, Sanjoy Chowdhury, Dinesh Manocha
arXiv AI
Aug 24

ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib

ProofJudge is an agentic large language model that evaluates the quality of formal proofs in Lean 4 beyond mere correctness. It scores proofs on five dimensions—library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions—using tool access to the relevant repository commit. The system was tested on 218 Mathlib declarations, achieving alignment with human reviewers between 63.5% and 80.8% and releasing its harness, dataset, and traces for open research.

By Shane Caldwell
arXiv AI
Aug 24

ExpertIVS: Sociological Expert Driven Individual Value Simulation in Large Language Models

ExpertIVS is a framework that uses 14 sociological expert agents to interpret World Values Survey responses, reconstructing individual value systems in a coherent, internally consistent manner rather than simply concatenating survey answers. It introduces a multi‑agent debate mechanism to assess LLM alignment with these value profiles during dynamic interactions. Experiments on 480 individuals from 12 countries show a 90.78% value restoration fidelity and a 5.3% improvement in value generalization over baseline methods, while also demonstrating strong personality discriminability and behavioral consistency.

By Zhen Wang, Yuqi Ren, Yuehan Cui, Hongxiang Wang, Jianxiang Peng, Zhaoxia Zhang, Bingkun Zhu, Tongxuan Zhang, Dezhi Tong, Deyi Xiong
arXiv AI
Aug 24

FL-MAESTRO: Multi-Agent LLM Orchestration for Resource-Constrained Federated Learning

FL-MAESTRO is a multi‑agent orchestrator that uses three specialized large language model agents to jointly decide the communication topology, per‑client resource allocation, and aggregation rule in each federated learning round. A coordinator merges the agents’ analyses, and a non‑LLM feasibility check validates the decision before execution. By filtering out clients whose updates would never be aggregated, the system eliminates the main source of wasted round energy in volatile edge networks and works across heterogeneous device classes without per‑class energy models, achieving comparable accuracy to the best energy‑aware baseline while reducing wasted energy from over a third to near zero on a non‑IID CIFAR‑10 benchmark.

By Jiajun Wu, Zirui Wang, Jiayu Zhou, Qiang Ye, Steve Drew
arXiv AI
Aug 24

TRACE: Training-time Report-guided and Clinically Ordered Concept Editing

TRACE is a training-time framework that uses structured radiology reports to guide concept editing, allowing image-only diagnosis during inference. It refines image-derived concepts with a teacher-guided editing mechanism in a malignancy-aware ordered concept space and introduces Strategic Concept Missing Training to handle incomplete annotations. The authors also present BUSC, a benchmark linking images, labels, and structured attributes, and show that TRACE outperforms existing methods on multiple datasets with better cross-domain robustness.

By Wentao Yue, Tianyou Lai, Jiayu Luo, Qingyu Mao, Ziying Wang, Zhenyuan Ning, Qilei Li
arXiv AI
Aug 24

WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving

arXiv:2608.20974v1 Announce Type: cross Abstract: Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent featu...

By Xinlin Wang, Yujiao Xiang, Yuheng Zhou, Jingqi Wang, Minqing Huang, Jiajie Huang, Dongxu Wei, Tingguang Zhou, Xiyang Wang, Gong Chen, Zhi Xu, Feiyang Tan, Hangning Zhou, Mu Yang
arXiv AI
Aug 24

ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents

ClawSentry is an open‑source, framework‑agnostic security supervision gateway designed to protect autonomous large language model (LLM) agents from progressive risks that can arise at four points in the agent control loop: skill admission, invocation‑time intent, execution‑time effect, and post‑action consequence. It introduces a multi‑tier decision engine—deterministic L1, rule‑anchored L2, and read‑only L3—alongside a First‑Use Skill Package Review (FSPR) and an Agent Harness Protocol (AHP) that applies a single policy across multiple agent runtimes without modifying their internals. Evaluation on SkillInject and the SkillsSafety benchmark shows that ClawSentry significantly reduces contextual adversarial skill risk (ASR) while maintaining high task success rates (TSR).

By Kai Wang, Zeming Wei, BiaoJie Zeng, Chang Jin, An Wang, Xiaokun Luan, Zhixiao Lin, Jingjing Qu, Xia Hu, Xingcheng Xu
arXiv AI
Aug 24

Can you see how I learn? Human observers' inferences about Reinforcement Learning agents' learning processes

The paper investigates how human observers interpret the learning processes of reinforcement learning (RL) agents. Using a novel observation-based paradigm, the authors conducted two experiments: an exploratory interview study with nine participants that identified four core themes—Agent Goals, Knowledge, Decision Making, and Learning Mechanisms—and a confirmatory study with 34 participants that applied the paradigm across navigation and manipulation tasks and two RL algorithms. Analyses of 816 responses validated the paradigm’s reliability and refined the thematic framework, showing how these themes evolve over time and interrelate.

By Bernhard Hilpert, Muhan Hou, Kim Baraka, Joost Broekens
arXiv AI
Aug 24

Recognizing Artificial Minds: A Philosophical Defense of AI Cognition

The paper defends the 'Whole Hog Thesis', arguing that sophisticated large language models such as ChatGPT are full linguistic and cognitive agents, possessing understanding, beliefs, desires, knowledge, and intentions. It rejects low‑level computational starting points and instead builds its case from high‑level behavioral observations, using Holistic Network Assumptions to link actions to mental states. The authors systematically rebut common objections—such as hallucinations and planning errors—by showing these resemble human fallibility and by challenging the necessity of traditional conditions like embodiment or semantic grounding.

By Herman Cappelen, Josh Dever
arXiv Computer Vision
Aug 24

GS-Net: Heterogeneous Vehicle Data Reuse via Generalizable Plug-and-Play 3DGS Module

GS‑Net is a lightweight plug‑and‑play module that expands sparse Structure‑from‑Motion point clouds into dense Gaussian primitives, enabling cross‑sensor view synthesis for autonomous driving. It learns a generalizable initialization for 3D Gaussian Splatting, improving rendering quality for both interpolated and extrapolated camera viewpoints. The authors introduce CARLA‑NVS, a benchmark with 12 uniformly spaced cameras, and show that GS‑Net outperforms standard 3DGS by 2.08 dB PSNR on interpolated views and 1.86 dB on extrapolated views while being 50× faster to initialize.

By Yichen Zhang, Zihan Wang, Jiali Han, Peilin Li, Jiaxun Zhang, Jianqiang Wang, Lei He, Keqiang Li
arXiv AI
Aug 24

AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale

AgentMercury is a scalable framework that synthesizes executable environments from high‑level business scenarios instead of task‑specific benchmarks. It creates a persistent world with entities, services, tools, and invariants, allowing diverse tasks and interaction trajectories to emerge naturally. The authors generated 4,783 environments across 14 industries and 50 countries, and training reinforcement‑learning agents on them improved performance on enterprise workflows and out‑of‑domain benchmarks, while the construction process itself can be learned to increase authoring success.

By Minbyul Jeong, Chanwoong Yoon
arXiv AI
Aug 24

Peer-Voted LLM-Agent Stress Tests Find Feed-Induced Lexical Convergence but No Reliable Matched-Exposure Advantage for Distributed Sources

The study introduces PV‑SST, a peer‑voted social‑platform testbed, and conducts a preregistered matched‑exposure experiment across four topics, four seeds, four model families, and three larger variants, totaling 448 trials. Results show that feeding agents a ranked list of prior peer posts increases lexical similarity in both the core panel and larger variants, but does not produce a reliable advantage in opinion alignment or survival rates. The only robust finding is lexical convergence driven by the peer‑ranked feed, with no consistent coordination benefit across models or topics.

By Rana Muhammad Usman, Dominic Williamson