Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,964 stories · RSS feed

arXiv Machine Learning

Jul 9

Activation Quantization of Vision Encoders Needs Prefixing Registers

arXiv:2510. 04547v5 Announce Type: replace Abstract: Large pretrained vision encoders are central to multimodal intelligence, powering applications from on-device vision processing to vision-language models.

By Seunghyeon Kim, Taesun Yeom, Jinho Kim, Wonpyo Park, Kyuyeun Kim, Jaeho Lee

llms efficiency multimodal

arXiv AI

Jul 9

When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs

arXiv:2607. 07395v1 Announce Type: cross Abstract: Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence.

By Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe, Vinti Agarwal, Pranav Murthy Yeluripaty

llms rag multimodal benchmarks

arXiv AI

Jul 9

Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

arXiv:2607. 06611v1 Announce Type: cross Abstract: Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words.

By Andrei-George Durdun, Victor Constantinescu, Radu Tudor Ionescu

llms nlp efficiency multimodal benchmarks

arXiv Machine Learning

Jul 9

InferNet: Exploiting Aggregate GPU Profiles as Side-Channel for DNN Architecture Inference

arXiv:2304. 03388v2 Announce Type: replace Abstract: Deep Neural Networks (DNNs) have become ubiquitous for their ability to solve problems across various domains, including computer vision, natural language processing, and speech recognition.

By Raja Hasnain Anwar, Jonah O'Brien Weiss, Tiago Alves, Sandip Kundu

llms computer-vision nlp robotics multimodal safety

arXiv Machine Learning

Jul 9

POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shaking

arXiv:2607. 06649v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on cross-modal tasks by jointly training on large-scale textual and visual data, where privacy-sensitive examples could be unintentionally encoded, raising concerns about privacy or copyright violation.

By Zhangheng LI, Jianing Zhu, Junyuan Hong, Sungmin Eum, Shuowen Hu, Suya You, Zhangyang Wang

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 9

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

arXiv:2607. 07189v1 Announce Type: new Abstract: Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the physics and inverse problems that underlie computational imaging.

By Ethan Chung, Chuanjun Zheng, Jasper Tan, Jingxi Li, Haopeng Zhang, Huaijin Chen

llms agents multimodal benchmarks

arXiv AI

Jul 9

On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

arXiv:2607. 07375v1 Announce Type: cross Abstract: Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems.

By Chethan Krishnamurthy Ramanaik, Tobias Callies, Michael Hecht, Eirini Ntoutsi

llms multimodal safety

arXiv AI

Jul 9

A Study of Commonsense Reasoning over Visual Object Properties

arXiv:2508. 10956v3 Announce Type: replace-cross Abstract: Inspired by human categorization, visual reasoning about object properties, such as physical attributes and functions, involves identifying and recognizing low-level details and higher-level abstractions.

By Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang, Floris den Hengst, Filip Ilievski

llms nlp multimodal benchmarks

arXiv AI

Jul 9

ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

arXiv:2607. 06633v1 Announce Type: cross Abstract: In this paper, we address the problem of multimodal federated learning with missing modality.

By Aavash Chhetri, Bibek Niroula, Eduard Vazquez, Yash Raj Shrestha, Prashnna Gyawali, Loris Bazzani, Binod Bhattarai

multimodal benchmarks

arXiv AI

Jul 9

Behavior Foundations for Quadruped Robots: ABot-C0 Technical Report

arXiv:2607. 07370v1 Announce Type: cross Abstract: In embodied intelligence systems, the motion controller serves as the critical bridge between semantic reasoning and physical execution.

By Xufeng Zhao, Fuzhi Yang, Jianhui Chen, Li Gao, Zhang Meng, Jie Gao, Yao Zheng, Wenyu Liu, Menglin Yang, Minqi Gu, Yaru Zhao, Honglin Han, Shihui Su, Zixiao Tang, Liu Liu, Mu Xu, Yang Cai, Wenbin Tang

agents robotics multimodal safety data-engineering

arXiv Machine Learning

Jul 9

Rethinking Multimodal Time-Series Forecasting Evaluation

arXiv:2607. 06973v1 Announce Type: new Abstract: We introduce a new context-enriched, multimodal time series forecasting benchmark, TimesX.

By Haoxin Liu, Yichen Zhou, Rajat Sen, B. Aditya Prakash, Abhimanyu Das

multimodal benchmarks

arXiv Machine Learning

Jul 9

Pelican-VLA 0.5: Attending Before Acting Benefits Generalization

arXiv:2607. 06655v1 Announce Type: cross Abstract: In this report, we present Pelican-VLA 0.

By Zeyuan Ding, Wenhai Liu, Yang Xu, Jiayu Hu, Yinda Chen, Yi Zhang, Yong Dai, Jian Tang, Xiaozhu Ju

computer-vision robotics fine-tuning multimodal

arXiv Machine Learning

Jul 9

Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector

arXiv:2607. 07382v1 Announce Type: new Abstract: Fast Radio Bursts (FRBs) are millisecond-duration radio transients whose automated detection increasingly relies on highly specialized deep learning models.

By Raiff H. Santos, Amilcar R. Queiroz, Tharcisyo S. S. Duarte, K. E. L. de Farias, Rafael A. Batista

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 9

Tree-of-Thoughts Reasoning for Text-to-Image In-Context Learning

arXiv:2607. 07117v1 Announce Type: cross Abstract: In text-to-image in-context learning (T2I-ICL), a model has to infer a latent compositional pattern from fewshot demonstrations for generating a query image.

By Stepanida Alekseeva, Jenifer Kalafatovich, Seong-Whan Lee

llms diffusion fine-tuning multimodal benchmarks

arXiv AI

Jul 9

VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting

arXiv:2507. 05116v5 Announce Type: replace-cross Abstract: Recent large-scale Vision Language Action (VLA) models have shown superior performance in robotic manipulation tasks guided by natural language.

By Juyi Lin, Amir Taherin, Arash Akbari, Arman Akbari, Lei Lu, Guangyu Chen, Taskin Padir, Xiaomeng Yang, Weiwei Chen, Yiqian Li, Xue Lin, David Kaeli, Pu Zhao, Yanzhi Wang

robotics efficiency multimodal benchmarks

arXiv AI

Jul 9

Heterogeneity-Adaptive Diffusion Schrodinger Bridge for PET-Guided Whole-Body MRI Translation

arXiv:2607. 07401v1 Announce Type: cross Abstract: While whole-body multimodal medical imaging scanners have been increasingly recognized for more effective medical applications, the excessive long acquisition time in PET-MR scanning is a major obstacle in more efficient clinical practice.

By Chengbo Wang, Jiacheng Yu, Linjie Bian, Ming Qi, Xiaosheng Liu, Tongtong Che, Jichang Zhang, Shuyu Li, Shaoli Song, Xiuying Wang

llms rag diffusion multimodal

arXiv Machine Learning

Jul 9

Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

arXiv:2607. 06875v1 Announce Type: cross Abstract: Understanding and forecasting audience reactions to video content are crucial for improving content creation, recommendation systems, and media analysis.

By Trang Nguyen, Sidong Zhang, Shiv Shankar, Gauri Jagatap, Deepak Chandran, Andrea Fanelli, Madalina Fiterau

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jul 9

Latent graph encoding of multimodal neuroimaging features with generative AI architectures

arXiv:2607. 07027v1 Announce Type: cross Abstract: While generative models enable encoding of complex neuroimaging data for feature generation and reconstruction, developing optimal architectural frameworks with appropriate encoding and latent space processes is crucial for studying structural and functional properties of the brain.

By Ishaan Batta, Meenu Ajith, Vince Calhoun

llms diffusion multimodal safety

arXiv Machine Learning

Jul 9

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

arXiv:2607. 06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs).

By Chongyu Fan, Pengfei Liu, Jingjia Huang, Sijia Liu, Yi Lin

llms reinforcement-learning multimodal

arXiv Machine Learning

Jul 9

Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices

arXiv:2607. 06651v1 Announce Type: new Abstract: Federated learning (FL) over mobile and edge devices increasingly involves multimodal models in which clients differ in both sensing capability and computational capacity.

By Quoc Bao Phan, Tuy Tan Nguyen

multimodal

Get the daily briefing