Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,626 stories · RSS feed

arXiv AI

Jun 4

EvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation

arXiv:2603. 09493v2 Announce Type: replace-cross Abstract: The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge.

By Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu, Zhenyu Liu, Yang Li

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jun 4

CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation

arXiv:2606. 03564v2 Announce Type: replace-cross Abstract: Reasoning segmentation aims to segment target objects described by complex language through joint visual-textual reasoning.

By Yifan Cao, Xiaocui Yang, Faxian Wan, Shi Feng, Daling Wang, Yifei Zhang

llms computer-vision multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Stateful Visual Encoders for Vision-Language Models

arXiv:2606. 04433v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes.

By Zirui Wang, Junwei Yu, Adam Yala, David M. Chan, Joseph E. Gonzalez, Trevor Darrell

llms agents fine-tuning multimodal

arXiv Machine Learning

Jun 4

A Geometric View of Counterfactual Behavior: Interaction of Boundary Proximity and Local Support

arXiv:2606. 04209v1 Announce Type: new Abstract: Counterfactual explanations seek small, semantically meaningful changes to an input that alter a model's prediction, and are widely used to interpret and audit machine learning systems.

By Ioanna Gemou, Matteo Gamba, Randall Balestriero, Ritambhara Singh

multimodal

arXiv Machine Learning

Jun 4

Radiomic Feature Selection Using Gradient Loss of Deep Neural Network for Lung Cancer Stage Detection

arXiv:2606. 04453v1 Announce Type: cross Abstract: Radiomics enables extraction of quantitative imaging biomarkers from medical images and has become an important tool for computer-aided cancer diagnosis.

By Hina Shakir, Mohammad Mohatram, Javeed Hussain, Syed Rizwan Ali, Muhammad Irfan Memon

multimodal

arXiv Machine Learning

Jun 4

Long-Term and Short-Term Transistor Aging in Deep Neural Networks: Impact and Mitigation

arXiv:2606. 04266v1 Announce Type: cross Abstract: Deep neural networks (DNNs) are used in a variety of real-world applications including, for example, image classification and speech recognition.

By Alireza Sarmadi, Virinchi Roy Surabhi, Prashanth Krishnamurthy, Hussam Amrouch, Ramesh Karri, Farshad Khorrami

computer-vision efficiency multimodal

arXiv Machine Learning

Jun 4

LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

arXiv:2511. 12581v2 Announce Type: replace Abstract: Static IR drop analysis is a fundamental and critical task in the field of chip design.

By Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng

llms diffusion multimodal benchmarks

arXiv AI

Jun 4

P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

arXiv:2606. 03376v2 Announce Type: replace-cross Abstract: Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs).

By Ruipeng Zhang, Zhihao Li, Haozhang Yuan, C. L. Philip Chen, Tong Zhang

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 4

MapAgent: An Industrial-Grade Agentic Framework for City-scale Lane-level Map Generation

arXiv:2606. 04513v1 Announce Type: new Abstract: Lane-level maps are critical infrastructure for autonomous driving and lane-level navigation, yet constructing and maintaining standardized lane networks for hundreds of cities remains highly labor-intensive.

By Deguo Xia, Zihan Li, Haochen Zhao, Dong Xie, Yuyao Kong, Xiyan Liu, Jizhou Huang, Mengmeng Yang, Diange Yang

agents robotics efficiency multimodal

arXiv AI

Jun 4

NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

arXiv:2606. 04806v1 Announce Type: cross Abstract: LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior.

By Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar

llms agents multimodal benchmarks safety

arXiv AI

Jun 4

Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Survey

arXiv:2304. 10891v3 Announce Type: replace-cross Abstract: Transformer-based models are becoming a central paradigm in autonomous driving because they can capture long-range spatial dependencies, multi-agent interactions, and multimodal context across perception, prediction, and planning.

By Juan Zhong, Yuhang Shi, Zukang Xu, Xi Chen

llms agents robotics efficiency multimodal safety

arXiv AI

Jun 4

GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval

arXiv:2603. 22121v2 Announce Type: replace-cross Abstract: Video Corpus Moment Retrieval (VCMR) aims to retrieve both the correct video and its temporal segment corresponding to a natural-language query, a task that is especially challenging for multi-verb queries where temporal action ordering is critical.

By Yunzhuo Sun, Xinyue Liu, Yanyang Li, Nanding Wu, Linlin Zong, Xianchao Zhang, Wenxin Liang

llms multimodal benchmarks

arXiv AI

Jun 4

Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)

arXiv:2606. 04240v1 Announce Type: cross Abstract: Retrieval over visually-rich documents, pages that interleave text with figures, tables, and charts, is essential for multimodal retrieval-augmented generation, yet most retrievers still discard the visual channel.

By Jingbiao Mei

llms rag fine-tuning multimodal

arXiv AI

Jun 4

A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks

arXiv:2510. 24342v2 Announce Type: replace Abstract: Prior brain-AI alignment studies are typically constrained by specific inputs and tasks, limiting their ability to capture organizational properties across models with different modalities.

By Silin Chen, Yuzhong Chen, Caiwei Wang, Zifan Wang, Junhao Wang, Zifeng Jia, Keith M Kendrick, Tuo Zhang, Lin Zhao, Dezhong Yao, Tianming Liu, Xi Jiang

llms fine-tuning multimodal safety

arXiv AI

Jun 4

Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching

arXiv:2512. 03553v3 Announce Type: replace-cross Abstract: Content moderation remains a critical yet challenging task for large-scale user-generated video platforms, especially in livestreaming environments where moderation must be timely, multimodal, and robust to evolving forms of unwanted content.

By Wei Chee Yew, Hailun Xu, Sanjay Saha, Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Kanchan Sarkar, Zhenheng Yang, Danhui Guan

llms multimodal safety

arXiv Machine Learning

Jun 4

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

arXiv:2606. 04613v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited.

By Alessandro Gambetti, Qiwei Han, Cl\'audia Soares, Hong Shen

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers

arXiv:2606. 04678v1 Announce Type: new Abstract: End-to-end ASR systems typically use fixed-depth acoustic encoders at inference, making it difficult to trade additional test-time computation for improved recognition without training a larger model.

By Yacouba Kaloga, Shashi Kumar, Shakeel A. Sheikh, Driss Khalil, Petr Motlicek, Ina Kodrasi

llms rag multimodal

arXiv AI

Jun 4

VGGSounder: Audio-Visual Evaluations for Foundation Models

arXiv:2508. 08237v4 Announce Type: replace-cross Abstract: The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding.

By Daniil Zverev, Thadd\"aus Wiedemer, Ameya Prabhu, Matthias Bethge, Wieland Brendel, A. Sophia Koepke

multimodal benchmarks

arXiv AI

Jun 4

Continual Visual and Verbal Learning Through a Child's Egocentric Input

arXiv:2606. 05115v1 Announce Type: cross Abstract: Children learn the meanings of words from a continuous, temporally structured stream of egocentric experience.

By Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman, Brenden Lake, Mengye Ren

computer-vision multimodal benchmarks

Hugging Face Trending Papers

Jun 3

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information. Learning disentangled representations is a principled way to identify these underlying shared and unique factors that are hidden in observational data.

rag multimodal benchmarks safety

Get the daily briefing