Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,414 stories · RSS feed

arXiv Machine Learning

Jun 3

Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

arXiv:2606. 03087v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the ability of large language model, yet headline accuracy gains often conceal a hidden cost: previously solved problems quietly become unsolvable as training proceeds.

By Chuanyu Qin, Chenxu Yang, Qingyi Si, Naibin Gu, Peng Fu, Zheng Lin

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 3

When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

arXiv:2606. 03712v1 Announce Type: new Abstract: Graph Language Models (GLMs) have become a promising direction for adapting Large Language Models (LLMs) to graph learning tasks.

By Ding Zhang, Runtao Zhou, Wenqing Zheng, Rizal Fathony, Bayan Bruss, Chirag Agarwal

llms efficiency multimodal safety

arXiv AI

Jun 3

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

arXiv:2606. 02642v1 Announce Type: cross Abstract: Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination.

By Chenshuang Zhang, Kyeong Seon Kim, Chengxin Liu, Tae-Hyun Oh

llms multimodal benchmarks safety

arXiv AI

Jun 3

P\textsuperscript{2}-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

arXiv:2606. 03376v1 Announce Type: cross Abstract: Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs).

By Ruipeng Zhang, Zhihao Li, Haozhang Yuan, C. L. Philip Chen, Tong Zhang

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 3

TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendation

arXiv:2509. 09685v5 Announce Type: replace-cross Abstract: We present TalkPlayData 2, a synthetic dataset for multimodal conversational music recommendation generated by an agentic data pipeline.

By Keunwoo Choi, Seungheon Doh, Juhan Nam

llms agents multimodal data-engineering

arXiv AI

Jun 3

Efficient ASR Training with Conversations that Never Happened

arXiv:2606. 03957v1 Announce Type: cross Abstract: Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data.

By M\'at\'e Gedeon, P\'eter Mihajlik

llms multimodal benchmarks

arXiv AI

Jun 3

MemVerse: Multimodal Memory for Lifelong Learning Agents

arXiv:2512. 03627v2 Announce Type: replace Abstract: Despite rapid progress in large-scale language and vision models, AI agents still suffer from a fundamental limitation: they cannot remember.

By Junming Liu, Yifei Sun, Weihua Cheng, Haodong Lei, Yirong Chen, Licheng Wen, Xuemeng Yang, Daocheng Fu, Pinlong Cai, Nianchen Deng, Yi Yu, Shuyue Hu, Botian Shi, Ding Wang

agents efficiency multimodal safety

arXiv AI

Jun 3

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

arXiv:2605. 18740v4 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image.

By Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, Yaojie Lu

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 3

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.

By Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 3

From 'What' to 'How' and 'Why': Sharing LLM-Generated Retrospective Summaries of Older Adults' Passive Tracking Data with Remote Family Members

arXiv:2606. 03876v1 Announce Type: cross Abstract: With the growing prevalence of modern ubiquitous computing technologies, multi-modal tracking systems hold promise for providing timely awareness and reassurance to stakeholders such as remote family members (RFMs) of older adults, who play a central role in care coordination.

By Jiachen Li, Reina Szeyi Chan, Akshat Choube, Xiang Zhi Tan, Elizabeth Mynatt, Varun Mishra

llms agents multimodal

arXiv AI

Jun 3

Efficient Transformer-Based Localized Patch Sampling for Choroid Plexus Segmentation in Multiple Sclerosis

arXiv:2606. 03566v1 Announce Type: cross Abstract: Background: The lateral ventricle choroid plexus (LVCP) is gaining recognition as a key imaging biomarker for multiple sclerosis (MS) related to physical disability and neuroinflammation.

By Po-Jui Lu, Alessandro Cagol, Mario Ocampo-Pineda, Federico Spagnolo, Marina Mastantuono, Andreea-Alexandra Aldea, Jannis M\"uller, \"Ozg\"ur Yaldizli, Matthias Weigel, Lester Melie-Garcia, Roberta Magliozzi, Maria Pia Sormani, Ludwig Kappos, Jens Kuhle, Cristina Granziera

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jun 3

CoughSense: Five-Class Respiratory Disease Classification via Whisper Encoder Fine-Tuning and Dual-Encoder Cross-Attention Fusion with Balanced Contrastive Learning

arXiv:2606. 02998v1 Announce Type: new Abstract: Automated cough analysis offers a path to low-cost respiratory screening, but most existing work stops at binary COVID-19 detection.

By Nikhil Vincent

fine-tuning multimodal benchmarks

arXiv AI

Jun 3

EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

arXiv:2606. 02739v1 Announce Type: cross Abstract: Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation.

By Hui Li, Yangfan Gao, Junlin Shang, Changhao Jiang, Tao Gui, Qi Zhang, Xuanjing Huang

llms diffusion nlp efficiency multimodal benchmarks

arXiv Machine Learning

Jun 3

A Fast Screening Approach for High-dimensional Outcomes and High-dimensional Predictors

arXiv:2606. 03018v1 Announce Type: cross Abstract: Modeling interactions among multimodal, high-dimensional data is intrinsically challenging due to ultra-high dimensionality and complex dependence structure with high level noise.

By Hongju Park, Zhenyao Ye, Shuo Chen

multimodal safety

arXiv Machine Learning

Jun 3

GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations

arXiv:2606. 03180v1 Announce Type: cross Abstract: Vision-language models (VLMs) for radiology have emerged as a scalable paradigm by leveraging image-report pairs naturally produced in clinical workflows.

By Jonggwon Park, Seongeun Lee, Junhyun Park, Hannah Yun, Hyunwoong Kim, Sohyun Jeong, Hyewon Kang, Byungmu Yoon, Kyoyun Choi

llms rag computer-vision efficiency multimodal safety

arXiv AI

Jun 3

MUSE: A Unified Agentic Harness for MLLMs

arXiv:2606. 03005v1 Announce Type: cross Abstract: Despite rapid progress, multimodal large language models (MLLMs) still fail on tasks that humans solve effortlessly, such as navigating a grid maze from a screenshot or selecting the correct puzzle piece.

By Jianglin Lu, Hailing Wang, Xu Ma, Qihua Dong, Mingyuan Zhang, Yizhou Wang, Yun Fu

llms agents multimodal benchmarks

arXiv AI

Jun 3

PAND: Prompt-Aware Neighborhood Distillation for Lightweight Fine-Grained Visual Classification

arXiv:2602. 07768v3 Announce Type: replace-cross Abstract: Distilling knowledge from large Vision-Language Models (VLMs) into lightweight networks is crucial yet challenging in Fine-Grained Visual Classification (FGVC), due to the reliance on fixed prompts and global alignment.

By Qiuming Luo, Yuebing Li, Feng Li, Chang Kong

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 3

Large AI Models in Dental Healthcare: From General-Purpose Systems to Domain-Specific Foundation Models

arXiv:2606. 02914v1 Announce Type: new Abstract: Background: Oral diseases affect nearly 3.

By Sema Helali, Lina Abu Nadab, Sausan Alqawas, Alaa Abd-Alrazaq, Faleh Tamimi, Rafat Damseh

agents computer-vision multimodal benchmarks safety

arXiv AI

Jun 3

MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models

arXiv:2512. 05530v2 Announce Type: replace Abstract: Recently, multimodal large language models (MLLMs) have been widely applied to reasoning tasks.

By Chuang Yu, Jinmiao Zhao, Mingxuan Zhao, Yunpeng Liu, Xiujun Shu, Yuanhao Feng, Bo Wang, Xiangyu Yue

llms multimodal benchmarks safety

arXiv AI

Jun 3

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

arXiv:2606. 03988v1 Announce Type: new Abstract: Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable.

By Mahtab Bigverdi, Lindsey Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dangjoo Kim, Zelun Luo, Linda Shapiro, Ranjay Krishna

llms multimodal benchmarks

Get the daily briefing