Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,974 stories · RSS feed

arXiv AI

Sep 23

Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

arXiv:2609.22628v1 Announce Type: new Abstract: Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We iso...

By Nikhil Reddy Pottanigari, Sepideh Kharaghani, Saverio Vadacchino, Alejandro Posada, Ying Zhang

multimodal

arXiv AI

Sep 23

From Research Frontier to Laboratory Bench: Design of a Four-Tier Experimental Teaching System for Multimodal Medical Image Intelligent Diagnosis

arXiv:2609.22790v1 Announce Type: new Abstract: Undergraduate programmes in intelligent medical engineering are expanding, yet laboratory curricula lag behind the multimodal, long-tailed, and distrib...

By Dongjing Shan, Yamei Luo, Jin Li, Yong Luo

multimodal safety

arXiv AI

Sep 23

When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used

arXiv:2609.22910v1 Announce Type: new Abstract: Vision-language agents that crop and zoom are trained with rewards that credit a successful tool call, yet a successful call does not show that the mod...

By Kunyu Peng, Junming Liu, Ruiqi He, Qingzhuo Wang, Jianzhong Qi, Xianhui Liu

agents multimodal benchmarks

arXiv AI

Sep 23

FireWorldBench: Benchmarking Complex Physical World Intelligence through Coupled-Field Fire Dynamics

arXiv:2609.23064v1 Announce Type: new Abstract: Understanding the physical world requires more than object recognition, scene description, and short-term visual prediction, as real-world physical sys...

By Qiang Chen, Hao Guo, Huatai Zhu, Tairan Huang, Yichao Cao, Hongyan Xu, Keke Huang, Haifeng Li, Yi Chen, Xiu Su

llms agents multimodal benchmarks

arXiv AI

Sep 23

From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving

arXiv:2609.23130v1 Announce Type: new Abstract: Large language model (LLM) inference is evolving from an engine-local optimization problem into a distributed control problem involving reusable state,...

By Twinkll Sisodia

llms agents efficiency multimodal benchmarks

arXiv AI

Sep 23

Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

arXiv:2609.23860v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consum...

By Tianyou Jiang

llms efficiency multimodal

arXiv AI

Sep 23

DocMIDE: Learning Multi-Hop Implicit Derivation in Visually Rich Documents

arXiv:2609.24092v1 Announce Type: new Abstract: Real-world document processing systems rely on rigid, predefined schemas, yet critical target fields often lack direct visual counterparts on the page....

By Jeremy Cerwin Wang, Wai Kit Wong, Jeff Kai Tai Tang

llms fine-tuning multimodal benchmarks

arXiv AI

Sep 23

VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning

arXiv:2609.24362v1 Announce Type: new Abstract: Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language m...

By Hexiong Yang, Mingrui Chen, Jie Cao, Ran He

llms agents multimodal benchmarks

arXiv AI

Sep 23

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

arXiv:2609.24677v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to make predictions from numerical time-series histories and textual events. Yet accuracy alone cann...

By Jie Gong, Maowei Jiang, Zhiwei Liu, Yankai Chen, Guojun Xiong, Xue Liu, Min Peng, Qianqian Xie, Sophia Ananiadou

llms multimodal benchmarks

arXiv AI

Sep 23

FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulation

arXiv:2609.22538v1 Announce Type: cross Abstract: Large language model (LLM) planners can decompose natural-language instructions and select reusable robot skills, but choosing the correct skill does...

By Ajay Vikram Periasami, Xinyuan Luo, Haoyu Li, Xianyi Cheng

llms agents computer-vision robotics multimodal

arXiv AI

Sep 23

OmniEcho: Spatial Audio Understanding for Embodied Agents

arXiv:2609.23407v1 Announce Type: cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for em...

By Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong

agents robotics multimodal benchmarks

arXiv AI

Sep 23

WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics

arXiv:2609.24094v1 Announce Type: cross Abstract: Visual analytics (VA) enables sensemaking through interactive visualization, but effective analysis often requires experts to translate high-level in...

By Yutong Chen, Zhike Tang, Zhihao Mai, Zhihao Shuai, Danli Luo, Jing Xu, Weikai Yang

llms agents multimodal benchmarks

arXiv AI

Sep 23

TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens

arXiv:2605.16638v2 Announce Type: replace Abstract: Recent research has demonstrated that Universal Multimodal Embedding (UME) benefits significantly from Chain-of-Thought (CoT) reasoning. In this pa...

By Jianpeng Cheng, Xian Wu, Jiangfan Zhang, Wentao Bao, Chaitanya Ahuja, Shlok Kumar Mishra, Xuanming Cui, Hanchao Yu, Yang Gao, Fan Xia, Haixing Dai, Frankie Yuan, Zihao Wang, Xiaobing Chen, Qi Guo, Shaodan Zhai, Aashu Singh, Xiangjun Fan, Jun Xiao

llms rag multimodal benchmarks

arXiv AI

Sep 23

Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence

arXiv:2510.26915v2 Announce Type: replace-cross Abstract: While heterogeneous teams have typically been designed for well-specified missions with known semantics, generative intelligence, i.e., large...

By Zachary Ravichandran, Fernando Cladera, Ankit Prabhu, Jason Hughes, Carlos Nieto-Granda, Varun Murali, Camillo Taylor, George J. Pappas, Vijay Kumar

llms robotics multimodal

arXiv AI

Sep 23

Cognition to Control - Multi-Agent Learning for Human-Humanoid Collaborative Transport

arXiv:2603.03768v2 Announce Type: replace-cross Abstract: Full-stack human-robot collaboration (HRC) can become brittle when replacing a planner, partner model, coordination policy, or controller cha...

By Hao Zhang, Yisen Li, Ruize Geng, Yves Tseng, Yaru Niu, Ding Zhao, H. Eric Tseng

llms agents reinforcement-learning robotics multimodal

arXiv Machine Learning

Sep 23

Entropy Can Flow, or It Can Guide. Be Entropy. LEDFlow: Introducing Entropy-guided Generation Order into Uniform Discrete Flow

arXiv:2609.25131v1 Announce Type: new Abstract: Uniform discrete flow permits repeated updates at every generation position. While continued revision supports correction of wrong tokens, it also expo...

By Tung Sum Thomas Kwok, Yidong Ouyang, Yingjia Wan, Ying Nian Wu, Zhijiang Guo, Oscar Leong

diffusion multimodal benchmarks

arXiv Machine Learning

Sep 23

Brain-Inspired Hierarchical Modularity for General Continual Learning

arXiv:2609.25146v1 Announce Type: new Abstract: Continual learning, the ability to learn from sequential experience while retaining and adapting prior knowledge, is central to intelligent systems ope...

By Hongwei Yan, Kanglei Zhou, Qi Cheng, Weiyi Dong, Chunyan Lan, Guanglong Sun, Jun Zhou, Qian Li, Yi Zhong, Liyuan Wang

robotics multimodal

arXiv Machine Learning

Sep 23

Exposing Blind Spots in Deep Imbalanced Regression Evaluation

arXiv:2609.25152v1 Announce Type: new Abstract: Deep Imbalanced Regression (DIR) addresses a common failure mode of regression models: target distributions are highly non-uniform, causing models to p...

By Noah C. Puetz, Jens U. Brandt, Marc Hilbert, Elena Raponi, Thomas B\"ack, Thomas Bartz-Beielstein

multimodal benchmarks

arXiv Machine Learning

Sep 23

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

arXiv:2609.25809v1 Announce Type: new Abstract: Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly man...

By Yuanteng Chen, Qiwei Lai, Chen Tianqi, Peisong Wang, Yuantian Shao, Nanxin Zeng, Zhilei Liu, Chuangyi Li, Jing Liu, Jian Cheng

llms efficiency multimodal benchmarks

arXiv Machine Learning

Sep 23

Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesis

arXiv:2609.25411v1 Announce Type: cross Abstract: Classifier-free Guidance (CFG) is widely adopted in text-to-speech (TTS) systems to enhance generation quality and conditioning fidelity by interpola...

By Biel Tura Vecino, Yoach Lacombe, Julian Weber, Zbigniew {\L}atka, Haitong Zhang, Logan Hart, Eren G\"olge

rag multimodal

Get the daily briefing