The paper introduces a multimodal in‑context learning framework that uses contrastive demonstration modeling to align large language models’ responses with the required reasoning paths. By contrasting suboptimal and better responses and incorporating a response‑conditioned retrieval mechanism, the method explicitly guides models beyond surface imitation. Experiments on various multimodal tasks, especially visual question answering, show consistent performance gains.
By Mingbo Yang, Wenqiang Wang, Zhaolu Kang, Peng Chen, Yannan Chen, Sunshang Wang, Yan Xiao
arXiv:2411. 15041v2 Announce Type: replace Abstract: Advanced Multimodal Large Language Models (MLLMs) struggle with recent Knowledge-based Visual Question Answering (VQA) tasks, such as INFOSEEK and Encyclopedic-VQA, due to their limited and frozen knowledge scope, often leading to ambiguous and inaccurate responses.
By Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, Jin Ma, Ying Shan, Weiming Hu
arXiv:2512. 03438v3 Announce Type: replace Abstract: Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-based rewards computed based on the final answers.
By Reuben Tan, Baolin Peng, Zhengyuan Yang, Hao Cheng, Oier Mees, Theodore Zhao, Andrea Tupini, Isar Meijer, Qianhui Wu, Yuncong Yang, Lars Liden, Yu Gu, Sheng Zhang, Xiaodong Liu, Lijuan Wang, Marc Pollefeys, Yong Jae Lee, Jianfeng Gao
DocPilot is an agentic framework for long visual document understanding that orchestrates evidence seeking through page-level navigation and fine-grained localization, integrating visual inspection, code-based numerical reasoning, and cross-modal reasoning. It employs curriculum-based agentic training to enable small language models to learn complex document solving, progressing from single-ability learning to adaptive orchestration. Experiments on benchmarks such as MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA show that DocPilot achieves strong gains with frontier VLM backbones, and its 4B variant competes with much larger models, demonstrating effectiveness and generalization across diverse long-document reasoning tasks.
By Zhuowen Liang, Xiaotian Lin, Yuyu Luo, Haixun Wang, Nan Tang
arXiv:2607. 22643v1 Announce Type: new Abstract: Multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries with external knowledge, but most existing systems still retrieve directly from raw multimodal input over a flat evidence space.
By Tianyu Yang, Shir Simon, Zhenzhen Li, Minhao Cheng, Xiangliang Zhang
Selection-Based Structured Reasoning (SSR) is a framework that replaces free-form reasoning with the selection of pre-specified natural‑language reasoning candidates, allowing multimodal agents to choose from a set of reusable high‑level reasoning traces. By scoring these candidates in parallel using a shared context KV cache, SSR eliminates the need for an auxiliary task head and reduces inference cost. Experiments on seven multimodal search benchmarks with 2B and 4B models show that SSR maintains competitive success rates while cutting per‑turn reasoning latency by over 90% and overall inference latency by 28–54%.
By Feiyu Gavin Zhu, Xiaoyu Zhu, Jiqi Yang, Rui Yang, Arnab Kumar Mondal, Yancheng Wang, Xinke Deng, Jean Oh, Reid Simmons, Joerg Liebelt, Xiang Kong, Zhongyu Jiang
The paper introduces ProAction, a multimodal dataset of 10,000 samples comprising visual, audio, and text inputs across 12 daily-life scenarios, designed to support the Proactive Robot Action Reasoning (ProRobo) problem. It presents a two-stage human-in-the-loop annotation pipeline that incorporates appraisal and Theory-of-Mind considerations to generate cognitively grounded high-level action labels. The authors benchmark multimodal large language models and propose MMC2Act, showing that training on ProAction significantly improves proactive action reasoning compared to general-purpose models.
By Zhihao Gu, Kechao Zhu, Yuanfeng Wu, Mohan Liu, Ankit Kumar Shaw, ChenDong Hong, Xuanyu Chen, Dengchen Mei, Xu Tianyi, Lin Wang
arXiv:2607. 10463v1 Announce Type: new Abstract: Agentic retrieval-augmented generation (RAG) extends static RAG by allowing language models to iteratively reason, generate search queries, retrieve evidence, and predict answers.
By Varun Gandhi, Jaewook Lee, Shantanu Todmal, Franck Dernoncourt, Ryan Rossi, Zichao Wang, Andrew Lan
AgentVidBench is a new multi‑hop video question‑answering benchmark designed to evaluate spatial, temporal, and causal reasoning in multimodal large language models (MLLMs). Unlike existing tests that focus on simple scene queries or global summaries, AgentVidBench includes step‑by‑step solution traces to assess whether agents gather the necessary evidence to justify their answers. Experiments with 12 MLLMs show limited single‑turn performance, but integrating these models into agentic workflows improves both accuracy and trajectory scores, establishing AgentVidBench as a comprehensive testbed for future research on agentic video understanding.
By Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Kangwook Lee
arXiv:2609.15296v1 Announce Type: new
Abstract: Universal multimodal embedding (UME) learns unified representations across modalities, enabling a single model to support diverse retrieval tasks. Rece...
By Mingzhou Jiang, Peixi Wu, Hang Cheng, Yunhao Zhou, Biao Yang, Wei Yuan, Yun Li, Fan Yang, Wenwu Ou, Honghui He
The Pistis Technical Report introduces the Pistis model family, comprising 27B- and 9B-parameter multimodal large language models built on Qwen3.6 and Qwen3.5. The models are developed through a scalable post‑training framework that begins with large‑scale multimodal supervised fine‑tuning and then applies Interleaved Distillation and Reinforcement Learning (IDRL) to integrate on‑policy distillation and reinforcement learning within a single training loop. Two specialized variants—Pistis‑Thinking for deep multimodal reasoning and Pistis‑Agentic for long‑horizon planning, iterative reasoning, and tool use—are produced at both scales, and a system‑level method called Pistis‑Auto‑Harnessing (PAH) further improves inference harness performance without updating model parameters.
By Heyun Chen, Xiaohan Lan, Jiaxi Li, Zhilin Lu, Qi She, Weiwen Xu, Fei Yu, Yujie Zhong, Jinghuan Chen, Zijian Feng, Siyu Jiao, Yiheng Lin, Xinhao Wang, Sihan Yang, Jieyu You, Changbin Zhang, Hengyu Zhang, Xudong Zhang, Yunqing Zhao, Shuai Zheng
arXiv:2609.15606v1 Announce Type: cross
Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress on short video understanding yet remain limited on long videos due to the...
By Weixin Xu, Zhenyu Yang, Bing Wang, Shengsheng Qian, Changsheng Xu