Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,277 stories · RSS feed

arXiv Machine Learning

Jun 16

Semi-Supervised Speech Confidence Detection using Pseudo-Labelling and Whisper Embeddings

arXiv:2606. 16505v1 Announce Type: cross Abstract: Understanding speaker confidence is crucial in educational settings, as it can enhance personalised feedback and improve learning outcomes.

By Adam Wynn, Jingyun Wang, Xiangyu Tan

rag multimodal

arXiv Machine Learning

Jun 16

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities

arXiv:2606. 15514v1 Announce Type: cross Abstract: Robotic systems perceive the world through multiple input modalities -- including visual camera streams and natural language instructions -- and must select appropriate actions based on these signals.

By Hassan Ismkhan, Hamid Bouchahcia

rag reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 16

YTClickbait21K: Human-Annotated Multimodal Dataset for YouTube Clickbait Detection Across Diverse Channels and Content Categories

arXiv:2606. 14780v1 Announce Type: cross Abstract: Clickbait content on video-sharing platforms poses a significant challenge to information reliability, yet progress in automated detection has been constrained by the lack of large-scale, high-quality multimodal datasets.

By Md. Minhazul Islam, Md. Tanbeer Jubaer, Amith Khandakar, Shovon Sarker, Sumaiya Rahman, Md. Masum Mia, Mohamed Arselene Ayari, Hamed Noori

multimodal benchmarks

arXiv AI

Jun 16

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

arXiv:2606. 14777v1 Announce Type: cross Abstract: Many moments in the real world do not wait for a user to ask.

By Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Haowen Hou, Zheming Liang, Congcong Wang, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang

llms agents multimodal

arXiv Machine Learning

Jun 16

Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective

arXiv:2606. 14883v1 Announce Type: cross Abstract: Continual vision-language models are commonly addressed through sequential fine-tuning; however, although this paradigm enables adaptation to new environments (tasks), it inherently emphasizes the contribution of previously learned environments (tasks) at the expense of the stability required to preserve previously acquired knowledge.

By Salimeh Sekeh, Mary Wisell

llms fine-tuning multimodal

arXiv Machine Learning

Jun 16

KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation

arXiv:2606. 16256v1 Announce Type: cross Abstract: Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge.

By Mao-Lin Luo, Yi-Lin Zhang, Zi-Hao Zhou, Yankun Hong, Xialiang Tong, Mingxuan Yuan, Tong Wei, Min-Ling Zhang

llms computer-vision nlp fine-tuning multimodal

arXiv AI

Jun 16

Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference

arXiv:2606. 15782v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated strong capabilities in vision-language understanding and natural-language response generation.

By Pratheswaran Hariharan, Haiping Xu, Donghui Yan

llms rag multimodal safety

arXiv Machine Learning

Jun 16

A Multimodal Approach to Alzheimer's Diagnosis: Geometric Insights from Cube Copying and Cognitive Assessments

arXiv:2512. 16184v2 Announce Type: replace Abstract: Early and accessible detection of Alzheimer's disease (AD) remains a critical clinical challenge, and cube-copying tasks offer a simple yet informative assessment of visuospatial function.

By Jaeho Yang, Kijung Yoon

multimodal safety

arXiv AI

Jun 16

MatchLM2Lite: A Scalable MLLM-to-Lite Framework for Reproduced Content Identification

arXiv:2606. 14786v1 Announce Type: cross Abstract: Content moderation is critical for online video platforms to ensure content safety, protect creators, and sustain positive user experiences.

By Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Zirui Zhu, Kanchan Sarkar, Kun Xu

llms efficiency multimodal safety

arXiv Machine Learning

Jun 16

Manifold-Orthogonal Dual-spectrum Extrapolation for Parameterized Physics-Informed Neural Networks

arXiv:2603. 13751v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) have achieved notable success in modeling dynamical systems governed by partial differential equations (PDEs).

By Zhangyong Liang, Huanhuan Gao

diffusion fine-tuning multimodal benchmarks

arXiv AI

Jun 16

Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memes

arXiv:2606. 15307v1 Announce Type: cross Abstract: Hateful and propagandistic memes exploit the interplay between images and text to convey harmful intent that neither modality reveals alone.

By Mohamed Bayan Kmainasi, Mucahid Kutlu, Ali Ezzat Shahroor, Abul Hasnat, Firoj Alam

llms reinforcement-learning efficiency multimodal benchmarks

arXiv Machine Learning

Jun 16

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

arXiv:2606. 15160v1 Announce Type: cross Abstract: Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years.

By David Huang, Lianlei Shan

llms diffusion reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jun 16

MAGE-RAG: Multigranular Adaptive Graph Evidence for Agentic Multimodal RAG in Long-Document QA

arXiv:2606. 15906v1 Announce Type: cross Abstract: Long-document multimodal question answering requires a system to locate sparse evidence in long PDFs and integrate clues from text, tables, images, charts, and complex layouts.

By Yilong Zuo, Xunkai Li, Jing Yuan, Qiangqiang Dai, Hongchao Qin, Ronghua Li

rag agents nlp multimodal

arXiv AI

Jun 16

Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech

arXiv:2606. 14750v1 Announce Type: cross Abstract: Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding.

By Adarsh Arigala, Arjun Gangwar, S Umesh, Yova Kementchedjhieva

rag fine-tuning multimodal

arXiv Machine Learning

Jun 16

The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

arXiv:2606. 16028v1 Announce Type: new Abstract: Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models.

By Thomas Dittrich, Oliver Potocki, Philipp Grohs

fine-tuning multimodal

arXiv AI

Jun 16

X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining

arXiv:2606. 14752v1 Announce Type: cross Abstract: Modern Vision-Language-Action (VLA) models must bridge pretrained vision-language reasoning and precise continuous robot control.

By Xirui Kang, Yanpei Shi, Lucy Liang, Roy Gan, Dongxiu Liu, Pushi Zhang, Danpeng Chen, Xiaoyi Qin, Yinan Zheng, Jinliang Zheng, Hao Wang, Xianyuan Zhan, Hang Su

nlp robotics efficiency multimodal safety

arXiv AI

Jun 16

AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

arXiv:2606. 15057v1 Announce Type: cross Abstract: Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.

By Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang, Yevgeniy Vorobeychik

llms agents multimodal benchmarks

arXiv AI

Jun 16

Active Inference with a Self-Prior in the Mirror-Mark Task

arXiv:2604. 09673v2 Announce Type: replace-cross Abstract: The mirror self-recognition test evaluates whether a subject touches a mark on its own body that is visible only in a mirror, and is widely used as an indicator of self-awareness.

By Dongmin Kim, Hoshinori Kanazawa, Yasuo Kuniyoshi

llms multimodal

arXiv AI

Jun 16

Unified Multimodal Model for Brain MRI Imputation and Understanding

arXiv:2606. 16484v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) hold great potential for medicine, as they inherit knowledge from LLM and allow multiple data modalities to be integrated, analysed and interpreted in natural language.

By Zhiyun Song, Che Liu, Tian Xia, Avinash Kori, Wenjia Bai

llms rag multimodal safety

arXiv AI

Jun 16

FlowMPC: Improving Flow Matching policies with World Models

arXiv:2606. 16286v1 Announce Type: cross Abstract: Flow Matching (FM) is a powerful approach for behavior cloning in multimodal action spaces [Jiang et al.

By Chandon Hamel

diffusion robotics multimodal

Get the daily briefing