Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,414 stories · RSS feed

arXiv Machine Learning

Jun 3

GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations

arXiv:2606. 03180v1 Announce Type: cross Abstract: Vision-language models (VLMs) for radiology have emerged as a scalable paradigm by leveraging image-report pairs naturally produced in clinical workflows.

By Jonggwon Park, Seongeun Lee, Junhyun Park, Hannah Yun, Hyunwoong Kim, Sohyun Jeong, Hyewon Kang, Byungmu Yoon, Kyoyun Choi

llms rag computer-vision efficiency multimodal safety

arXiv Machine Learning

Jun 3

A Quantitative Approximation Framework for Flow Distillation in Diffusion Models

arXiv:2606. 03820v1 Announce Type: cross Abstract: We develop a quantitative approximation framework for diffusion distillation, viewing few-step sampling as error propagation under compositions of learned flow maps.

By Weiguo Gao, Ming Li, Lei Shi, Hanfei Zhou

diffusion efficiency multimodal

arXiv AI

Jun 3

\textsc{CR-Seg}: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation

arXiv:2606. 03564v1 Announce Type: cross Abstract: Reasoning segmentation aims to segment target objects described by complex language through joint visual-textual reasoning.

By Yifan Cao, Xiaocui Yang, Faxian Wan, Shi Feng, Daling Wang, Yifei Zhang

llms computer-vision multimodal benchmarks safety

arXiv Machine Learning

Jun 3

Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoning

arXiv:2606. 02842v1 Announce Type: new Abstract: Multimodal spatial reasoning often relies on long chains of intermediate textual and visual thoughts, where accumulating visual tokens and dense cross-modal attention incur substantial computation and memory overhead.

By Yixian Shen, Zhiheng Yang, Qi Bi, Changshuo Wang, Shuai Wang, Jia-Hong Huang, George Floros, Prayag Tiwari, Anuj Pathania

efficiency multimodal

arXiv Machine Learning

Jun 3

VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring

arXiv:2606. 03954v1 Announce Type: cross Abstract: As AI systems increasingly assist humans in physical tasks, ensuring safety becomes paramount -- physical actions carry immediate and irreversible consequences that digital errors do not.

By Hanjiang Hu, Yiyuan Pan, Jiaxing Li, Xusheng Luo, Alexander Robey, Na Li, Yebin Wang, Changliu Liu

agents robotics multimodal benchmarks safety

arXiv AI

Jun 3

AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task

arXiv:2606. 03967v1 Announce Type: cross Abstract: We describe AlignAtt4LLM, an IWSLT 2026 simultaneous speech translation system for English to German, Italian, and Chinese.

By Quentin Fuxa, Dominik Mach\'a\v{c}ek

llms multimodal safety

arXiv Machine Learning

Jun 3

Social Caption: Evaluating Social Understanding in Multimodal Models

arXiv:2601. 14569v2 Announce Type: replace-cross Abstract: Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions.

By Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe, Louis-Philippe Morency

llms multimodal

arXiv AI

Jun 3

Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining

arXiv:2509. 22468v2 Announce Type: replace-cross Abstract: High-quality molecular representations are essential for property prediction and molecular design, yet large labeled datasets remain scarce.

By Boshra Ariguib, Mathias Niepert, Andrei Manolache

llms rag diffusion fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 3

Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

arXiv:2606. 03100v1 Announce Type: cross Abstract: Recently, zero-shot 3D scene understanding via 2D Vision-Language Models (VLMs) has gained increasing research interest due to their promising spatial reasoning capabilities.

By Dongsheng Wang, Dawei Su, Hui Huang

llms rag nlp multimodal benchmarks

arXiv AI

Jun 3

Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement

arXiv:2412. 01282v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) bring powerful understanding and reasoning capabilities to multimodal tasks.

By Qianhan Feng, Wenshuo Li, Tong Lin, Xinghao Chen

llms rag efficiency multimodal benchmarks safety

arXiv AI

Jun 3

TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

arXiv:2606. 03626v1 Announce Type: cross Abstract: Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks.

By Chao Wen, Jacqueline Staub, Adish Singla

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 3

P\textsuperscript{2}-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

arXiv:2606. 03376v1 Announce Type: cross Abstract: Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs).

By Ruipeng Zhang, Zhihao Li, Haozhang Yuan, C. L. Philip Chen, Tong Zhang

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 3

AURA: Action-Gated Memory for Robot Policies at Constant VRAM

arXiv:2606. 02775v1 Announce Type: new Abstract: The KV-cache is the right memory for datacenters but the wrong memory for robots.

By Josef Chen

agents robotics multimodal benchmarks

arXiv AI

Jun 3

Decomposing how prompting steers behavior

arXiv:2606. 03093v1 Announce Type: new Abstract: Prompting steers large language models (LLMs) and vision-language models (VLMs) without weight updates, but it remains unclear how instruction changes reshape internal representations to produce behavior.

By Fan L. Cheng, Nikolaus Kriegeskorte

llms multimodal

arXiv AI

Jun 3

From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds

arXiv:2606. 03557v1 Announce Type: new Abstract: As generative AI capabilities expand, AI-driven virtual worlds face a growing architectural challenge.

By Louis Nisiotis, Aimilios Hadjiliasi

rag agents fine-tuning multimodal

arXiv AI

Jun 3

TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

arXiv:2605. 20731v2 Announce Type: replace-cross Abstract: Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison.

By Haonan Zhu, Elad Hirsch, Alexandria Minetti, Allison Nulty, Purvanshi Mehta

diffusion multimodal benchmarks safety

arXiv AI

Jun 3

PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

arXiv:2605. 27762v2 Announce Type: replace Abstract: We present PEAM, a Parametric Embodied Agent Memory framework in Minecraft that transforms agent memory from inference-time retrieval into parameter-resident skills internalized through experience.

By Yuchen Guo, Junli Gong, Weicheng Wang, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

llms agents robotics fine-tuning multimodal

arXiv AI

Jun 3

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

arXiv:2606. 03879v1 Announce Type: cross Abstract: As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design.

By Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 3

Multi-Modal Machine Learning for Breast Cancer Recurrence Prediction

arXiv:2606. 02892v1 Announce Type: new Abstract: Breast cancer recurrence, a leading cause of long-term mortality among survivors, requires timely and accurate risk assessment to guide follow-up care and treatment planning.

By Jiahao Shao, Xudong Wang, Anam Nawaz Khan, Christopher Brett, Xueping Li, Bing Yao

multimodal benchmarks

arXiv AI

Jun 3

WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation

arXiv:2503. 07265v4 Announce Type: replace-cross Abstract: Text-to-Image (T2I) models are capable of generating high-quality artistic creations and visual content.

By Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Fanqing Meng, Kunpeng Ning, Bin Zhu, Li Yuan

diffusion multimodal benchmarks safety

Get the daily briefing