arXiv AI By Haolin Yang, Wenshan Wu, Jin Jiang, Yu Cheng, Yan Xia, Ting Song, Yangyu Huang, Zhiliang Peng, Wenhui Wang, Hangbo Bao, Yuzhe Zhang, Wen-Da Wei, Tengchao Lv, Li Dong, Lei Cui, Furu Wei

Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding

Read the original on arXiv AI →

The paper introduces Vision-as-a-Tool (VaaT), a paradigm that treats multimodal large language models as external utilities for visual reasoning, allowing text-only agents to offload visual tasks and vision-capable agents to augment perception on demand. It presents VHarness, a unified framework with five atomic actions, and evaluates it on Image-to-HTML, Image-to-SVG, and Image-to-3D generation tasks. Results show consistent improvements in visual fidelity and geometric accuracy for both text-only and multimodal coding agents.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 24

A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications

The paper surveys how large multimodal models (LMMs) enhance agentic frameworks that combine perception, memory, reasoning, planning, and action. It examines the integration of multiple modalities—text, images, audio, and video—through delegated, late‑fusion, and early‑fusion architectures, and maps these designs to agent capabilities. The survey also reviews multimodal agentic systems in robotics, web navigation, multimedia content creation, and video understanding, evaluating performance, efficiency, and scalability trade‑offs.

By Neel Mokaria, Rishie Raj, Dheeraj Baiju, Xiaoqian Shen, Shraman Pramanick, Kevin Qinghong Lin, Arda Senocak, Mike Zheng Shou, Philip Torr, Mohamed Elhoseiny, Yapeng Tian, Ruohan Gao, Salman Khan, Sayan Nag, Sanjoy Chowdhury, Dinesh Manocha
arXiv AI
Jun 16

Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning

arXiv:2606. 15231v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios.

By Zhengbo Zhang, Changtao Miao, Jinbo Su, Zhaowen Zhou, Chunxia Zhang, Xukai Wang, Ruiqi Liu, Kaiyuan Zheng, Jiansheng Cai, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan
arXiv Machine Learning
Oct 15

How Far Can Coding Agents Go in Visual Generation?

The paper investigates general‑purpose coding agents that generate images by writing and executing programs, contrasting them with conventional generative models. On the AgentDrawBench benchmark, these agents excel at preserving structure and meeting precise visual constraints, achieving competitive performance in tasks where compositional accuracy outweighs photorealism, though they fall short in perceptual richness. The study further shows that the agents’ visual code can serve as an external language of thought, retaining substantial information for visual question answering and suggesting an internal, compositionally structured representation that guides visual operations across different drawing interfaces.

By Hengguang Zhou, Yuanhao Ban, Jiaqi Feng, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh
arXiv AI
Oct 16

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

CoWorld Reasoner is an agentic framework that leverages a Video World Model (VWM) as a unified visual reasoning engine for Vision‑Language Models (VLMs). It exposes multiple visual operations through tools and skills, allowing the VLM to decide when to imagine, select an operation, generate visual content, and interpret the resulting evidence. The framework includes a VLM‑VWM Harness Co‑Evolution mechanism that jointly refines tools, skills, and execution code based on feedback, while keeping model weights fixed. Across seven benchmarks covering spatial, temporal, causal, and perceptual reasoning, CoWorld Reasoner outperforms direct VLM reasoning and improves accuracy on all benchmarks, achieving notable gains on MMSI and ERQA.

By Dongjie Cheng, Ruifeng Yuan, Ran Wei, Ning Song, Yongqi Li, Wenjie Li