Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,869 stories · RSS feed

arXiv Machine Learning

Jul 31

S-GRPO: Unified Post-Training for Large Vision-Language Models

arXiv:2604. 16557v2 Announce Type: replace Abstract: Current post-training methodologies for adapting Large Vision-Language Models (LVLMs) generally fall into two paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

By Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu

llms reinforcement-learning fine-tuning multimodal

arXiv AI

Jul 31

Voice Memory for Agentic Speech Recognition

arXiv:2607. 26410v1 Announce Type: cross Abstract: We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.

By Chao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg

agents multimodal

arXiv Machine Learning

Jul 31

Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models

arXiv:2607. 27304v1 Announce Type: new Abstract: Medical vision-language models (VLMs) generate chain-of-thought (CoT) reasoning before answering clinical questions, but whether this reasoning causally influences predictions remains unclear.

By Supratik Bhowal, Subhrajyoti Basu, Aritra Gir Mahanta, Anik Pal Chowdhury

llms multimodal

arXiv AI

Jul 31

Anatomy Contextualized Adaption of CT Foundation Models

arXiv:2607. 27154v1 Announce Type: cross Abstract: CT vision-language foundation models have demonstrated promising performance across downstream tasks, but are typically trained with whole-volume representations that dilute fine-grained anatomical signals.

By Roshan Kenia, Stephanie L McNamara, William Lotter

llms rag multimodal safety

arXiv Machine Learning

Jul 31

FedOGL: Combating Catastrophic Forgetting in Federated Open-World Multimodal Graph Learning

arXiv:2607. 27665v1 Announce Type: new Abstract: Federated graph learning enables collaborative training over decentralized graph data without sharing raw graph information.

By Zekai Chen, Haodong Lu, Shihao Li, Weiwei Ji, Xunkai Li, Xun Wu, Yinlin Zhu, Rong-Hua Li

efficiency multimodal

arXiv Machine Learning

Jul 31

Context-Informed Ship Trajectory Prediction via Conditional Attention

arXiv:2607. 27418v1 Announce Type: new Abstract: Long-term ship trajectory prediction is a fundamental capability for maritime safety and autonomous navigation.

By Yuan Guan, Chandler Squires, Timothy Hu, Pradeep Ravikumar

llms agents multimodal safety

arXiv Machine Learning

Jul 31

ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs

arXiv:2607. 28538v1 Announce Type: cross Abstract: Classifying pathological scars from clinical photographs requires distinguishing keloids from hypertrophic scars despite limited expert-labeled data and substantial acquisition variation across hospitals.

By Ruman Wang, Hangting Ye

llms computer-vision multimodal

arXiv Machine Learning

Jul 31

Cross-Embodiment Transfer via Behavior-Aligned Representations

arXiv:2607. 27549v1 Announce Type: cross Abstract: Recent progress in large-scale imitation learning for robot manipulation has been driven by leveraging datasets across a wide range of robot embodiments.

By Ajay Sridhar, Jensen Gao, Jonathan Yang, Jean Mercat, Suneel Belkhale, Dorsa Sadigh

robotics multimodal benchmarks

arXiv AI

Jul 31

TANDEM: Temporal-Aware Neural Detection for Multimodal Hate Speech

arXiv:2601. 11178v3 Announce Type: replace Abstract: Social media platforms are increasingly dominated by long-form multimodal content, where harmful narratives are constructed through a complex interplay of audio, visual, and textual cues.

By Girish A. Koushik, Helen Treharne, Diptesh Kanojia

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 31

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

arXiv:2607. 27084v1 Announce Type: cross Abstract: Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers.

By Zihan Deng, Chuanzhi Xu, Huiqi Liang, Haoyang Li, Xiaozhen Zhong, Lequan Yu

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jul 31

Scalable Drift Monitoring in Medical Imaging AI

arXiv:2410. 13174v3 Announce Type: replace-cross Abstract: The integration of artificial intelligence (AI) into medical imaging has advanced clinical diagnostics but poses challenges in managing model drift and ensuring long-term reliability.

By Jameson Merkow, Felix J. Dorfner, Xiyu Yang, Alexander Ersoy, Giridhar Dasegowda, Mannudeep Kalra, Matthew P. Lungren, Christopher P. Bridge, Ivan Tarapov

rag multimodal

arXiv AI

Jul 31

Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

arXiv:2607. 25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases.

By Mateusz Koz{\l}owski

llms multimodal benchmarks

arXiv AI

Jul 31

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

arXiv:2607. 26735v1 Announce Type: cross Abstract: Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering.

By Xiaolong Liu, Junjian Li, Yuan Xiao, Jiaqi Deng, Dayong Ye, Tianqing Zhu, Huan Huo

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 31

Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.

By Stephen Gould, Anton van den Hengel

llms efficiency multimodal benchmarks

arXiv AI

Jul 31

Visual Credit Audit for Multimodal Spatial Reasoning

arXiv:2607. 27069v2 Announce Type: cross Abstract: Closed yes/no spatial benchmarks can reward a correct answer even when the image adds little support beyond no-image contexts.

By Feixiang Liu, Qiang Qiu, Lanbo Sun, Nan Wei, Huawei Shen, Xueqi Cheng

multimodal benchmarks safety

arXiv Machine Learning

Jul 31

Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learning

arXiv:2607. 27766v1 Announce Type: cross Abstract: On-device in-context learning (ICL) relies on pre-inference retrieval to select demonstrations for useful context before downstream model inference.

By Xinyu Luo, Hui Liu, Yihua Shao, Junyi Yang, Arindam Basu, Haoliang Li

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jul 31

TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification

arXiv:2607. 27289v1 Announce Type: new Abstract: The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction.

By Yu Chang, Anzhe Cheng, Chenwei Wu, Zhuoran Wang, Jiahao Chen, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Paul M. Thompson, Liyue Shen, Paul Bogdan

multimodal benchmarks

arXiv Machine Learning

Jul 31

ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

arXiv:2607. 28627v1 Announce Type: cross Abstract: Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints.

By Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem

llms rag efficiency multimodal benchmarks

arXiv AI

Jul 31

Hearsay: Vision-Language Medical Diagnoses Without an Image

arXiv:2607. 26886v1 Announce Type: cross Abstract: When asked to describe a medical image that was never attached, frontier vision-language models do not abstain: they confabulate a diagnosis.

By Siddharth Vohra

llms multimodal

arXiv AI

Jul 31

Facial-Expression-Aware Prompting for Empathetic LLM Tutoring

arXiv:2604. 15336v2 Announce Type: replace-cross Abstract: Large language models (LLMs) enable increasingly capable tutoring-style conversational agents, yet effective tutoring requires sensitivity to learners' affective and cognitive states beyond text alone.

By Shuangquan Feng, Laura Fleig, Ruisen Tu, Philip Chi, Edmund Bu, Melinda Ozel, Junhua Ma, Teng Fei, Virginia R. de Sa

llms agents multimodal

Get the daily briefing