The paper introduces ProCoRe, a contrastive reinforcement learning approach that improves visual grounding in multimodal large language models (MLLMs) for in-context learning (ICL). By generating captions for contrastive image pairs, decomposing them into propositions, and optimizing contrastive rewards, ProCoRe enhances alignment with relevant visual evidence without requiring ICL-formatted training data. The method achieves significant gains in multimodal ICL classification accuracy and captioning ROUGE-L over the Qwen-VL-3-8B baseline and outperforms state‑of‑the‑art ICL‑data‑dependent methods across multiple benchmarks. Additionally, the authors present PICL, a personalized multimodal ICL captioning benchmark designed to test genuine understanding of visual demonstrations.
By Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi
LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for addit...
TwinICL is a procedurally generated benchmark that pairs matched text and image versions of tasks to enable controlled comparison of in‑context learning (ICL) across modalities. Experiments on six open‑weight models and 38 tasks show that multimodal ICL consistently underperforms text‑only ICL, with varying gaps by task family. Interventions targeting visual access, task framing, and reasoning can recover strong multimodal performance on a diagnostic subset, yet a modality gap remains even when explicit task instructions are provided, highlighting the dual role of demonstrations as context and evidence.
By Zihan Xue, Po-Yi Lu, Serhii Honcharenko, Zih-Ching Chen, Hsuan-Tien Lin, Nanyun Peng, I-Hung Hsu, Kuan-Hao Huang
arXiv:2609.19138v1 Announce Type: new
Abstract: Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations...
By Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu
arXiv:2607. 13854v2 Announce Type: replace Abstract: Multimodal agents that think with images iteratively manipulate visual evidence and invoke tools across many steps.
By Ru Zhang, Weijie Qiu
VideoGen-Agent is a multimodal agent that uses multitask agentic reinforcement learning to coordinate external tools for video generation. It learns to augment, generate, and verify videos through multi‑turn interactions, guided by prompts and intermediate observations. On the new VABench benchmark, the agent improves base text‑to‑video performance by 19.1 points, and further upgrades to generation tools raise the score to 86.1, with human raters favoring the upgraded configuration in 84.3% of comparisons.
By Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang, Zihao Lin, Kaituo Feng, Suozhi Huang, Xiangyi Li, Yu Li, Chunyuan Li, Shilong Liu, Mengdi Wang
arXiv:2609.40048v1 Announce Type: new
Abstract: Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget,...
By Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen
The paper surveys emerging video understanding agents that use video as the primary source and actively decide what information to inspect, retain, verify, and act upon. It formalizes an agent loop for video understanding and organizes the literature through a challenge‑to‑design taxonomy linking context bottlenecks, evidence sparsity, temporal causality, and multimodal ambiguity to corresponding design solutions. The survey also reviews state‑space paradigms, learning paradigms, supervision signals, benchmarks, evaluation protocols, and outlines open directions toward agentic‑native temporal modeling and video‑native agents.
By Xinyu Deng, Siwen Luo, Daochang Liu
The paper introduces AVA, a training‑free framework that improves interleaved multimodal in‑context learning for frozen large vision‑language models. AVA decomposes the interleaved context into per‑sample threads, selects influential demonstrations while removing redundancy, and calibrates consistency using a weak sibling model’s confidence. Across five vision‑language benchmarks, AVA raises performance by up to 2.4 points, narrows the visual performance gap on ScienceQA‑IMG, and remains robust even with 30% noisy demonstrations, all while keeping the LVLM frozen and adding only modest latency.
By Yigit Yilmaz, Elena Petrova, Mehmet Kaya, Lucia Rossi, Amir Rahman
VISTA is a method for active multimodal agents that internalizes collective visual experience via on‑policy distillation. It turns observations from multiple rollouts of the same input into shared supervision, using Collective Visual Experience Distillation (CVED) to organize observations with context and Heterogeneity‑Aware Policy Improvement (HAPI) to reinforce successful trajectories and guide learning from unsuccessful ones. The approach lets an experience‑conditioned teacher evaluate a student’s partial responses, enabling discoveries from one trajectory to inform others without altering the student’s original history, and achieves superior performance on fine‑grained perception and general reasoning tasks compared to comparable agents.
By Zheng Jiang, Houde Qian, Yiming Chen, Ling Li, Chaoyang Li, Yueqi Li, Yuxuan Liu, Lifeng Sun
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, sp...
arXiv:2606. 00054v1 Announce Type: cross Abstract: Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models.
By Zhiyuan Feng, Qixiu Li, Huizhi Liang, Rushuai Yang, Yichao Shen, Zhiying Du, Zhaowei Zhang, Yu Deng, Li Zhao, Hao Zhao, Zongqing Lu, Oier Mees, Marc Pollefeys, Jiaolong Yang, Baining Guo