Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv AI

Jun 2

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

arXiv:2606. 01703v1 Announce Type: cross Abstract: We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions.

By Jiashuo Yu, Yao Yao, Boyu Chen, Alex Wang

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

PATRA: Pattern-Aware Alignment and Balanced Reasoning for Time Series Question Answering

arXiv:2602. 23161v4 Announce Type: replace Abstract: Time series reasoning demands both the perception of complex dynamics and logical depth.

By Junkai Lu, Peng Chen, Xingjian Wu, Yang Shu, Chenjuan Guo, Christian S. Jensen, Bin Yang

llms nlp multimodal safety

arXiv Machine Learning

Jun 2

Towards Sparse Video Understanding and Reasoning

arXiv:2602. 13602v2 Announce Type: replace-cross Abstract: We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA).

By Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

llms agents nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

arXiv:2605. 23500v2 Announce Type: replace-cross Abstract: Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis.

By Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms agents computer-vision reinforcement-learning multimodal benchmarks

arXiv AI

Jun 2

APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

arXiv:2601. 21444v2 Announce Type: replace-cross Abstract: The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs).

By Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu

rag multimodal

arXiv AI

Jun 2

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

arXiv:2601. 06199v3 Announce Type: replace-cross Abstract: Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens.

By Junseok Lee, Sangyong Lee, Chang-Jae Chun

llms multimodal benchmarks

arXiv AI

Jun 2

STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models

arXiv:2606. 01790v1 Announce Type: cross Abstract: Vision-language-model-based graphical user interface (GUI) agents have shown broad automation capabilities, yet deployment is bottlenecked by a key-value (KV) cache that grows linearly with interaction steps.

By Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 2

ProductWebGen: Benchmarking Multimodal Product Webpage Generation

arXiv:2606. 01022v1 Announce Type: cross Abstract: Crafting a product display webpage from a source product image, along with layout and visual content instructions, holds significant practical value for domains such as marketing, advertising, and E-commerce.

By Zhihong Liu, Siqi Kou, Zheng Li, Ye Ma, Quan Chen, Peng Jiang, Kai Yu, Zhijie Deng

llms diffusion fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

arXiv:2606. 00535v1 Announce Type: new Abstract: Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored.

By Zining Liu, Yunhai Hu, Tianhua Xia, Bo Bao, Eric Sather, Vithursan Thangarasa, Sai Qian Zhang

llms efficiency multimodal

arXiv AI

Jun 2

SilentDrift: Exploiting Action Chunking for Stealthy Backdoor Attacks on Vision-Language-Action Models

arXiv:2601. 14323v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are increasingly deployed in safety-critical robotic applications, yet their security vulnerabilities remain underexplored.

By Bingxin Xu, Yuzhang Shang, Binghui Wang, Emilio Ferrara

rag robotics multimodal safety

arXiv AI

Jun 2

Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty

arXiv:2606. 00670v1 Announce Type: cross Abstract: Face-to-face speech comprehension is inherently multimodal, integrating acoustic signals with visible articulation, facial expression, head motion, and other socially relevant cues.

By Zhou Yang, Yueyi Yang

multimodal

arXiv AI

Jun 2

DiffCrossGait: Trajectory-Level Alignment for 2D-3D Cross-Modal Gait Recognition via Latent Diffusion

arXiv:2606. 00153v1 Announce Type: cross Abstract: Cross-modal 2D-3D gait recognition is impeded by inherent domain discrepancies between 2D silhouette and 3D LiDAR range-view representations.

By Zhiyang Lu, Ming Cheng

rag diffusion multimodal benchmarks safety

arXiv AI

Jun 2

EVA-Net: Subject-Independent EEG Motor Decoding with Video-Derived Motor Priors

arXiv:2606. 01884v1 Announce Type: new Abstract: Practical non-invasive Brain-Computer Interface (BCI) systems require EEG decoders with strong cross-subject generalization and minimal calibration.

By Ziyuan Li, Yueyu Sun, Yimeng Zhang

efficiency multimodal benchmarks

arXiv AI

Jun 2

DarkVesselNet: Multi-Modal Remote Sensing and Trajectory Reasoning for Dark Vessel Detection

arXiv:2606. 00445v1 Announce Type: cross Abstract: Dark vessel detection requires fusing what vessels report through AIS with what satellites observe through radar and optical sensors.

By Arun Sharma

multimodal

arXiv AI

Jun 2

Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training

arXiv:2601. 23220v2 Announce Type: replace-cross Abstract: Despite recent Multimodal Large Language Models (MLLMs)' linguistic prowess in medical diagnosis, we find even state-of-the-art MLLMs suffer from a critical perceptual deficit: geometric blindness.

By Anglin Liu, Ruichao Chen, Yi Lu, Hongxia Xu, Jintai Chen

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

MMSkills: Towards Multimodal Skills for General Visual Agents

arXiv:2605. 13527v3 Announce Type: replace Abstract: Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines.

By Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

agents multimodal benchmarks

arXiv AI

Jun 2

Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space

arXiv:2606. 01909v1 Announce Type: cross Abstract: We present Echo, a proof-of-concept audio system built around a single 25 M-parameter ViT encoder.

By Louis Mouchon

rag diffusion fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

Capability and Robustness Cannot Both Be Free: An Information-Theoretic Bound for Vision-Language-Action Models

arXiv:2605. 25889v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models reach high success rates on clean inputs but collapse under small adversarial perturbations: a $16/255$ PGD attack drops OpenVLA-7B's LIBERO success from $95\%$ to under $5\%$.

By Jianwei Tai

multimodal safety

arXiv AI

Jun 2

SentimentLens: Reconciling Sentiment and Ratings via Dual-Modality in the Hospitality Sector

arXiv:2606. 00084v1 Announce Type: cross Abstract: Online travel platforms generate vast volumes of user-generated hotel reviews, offering rich opportunities to understand traveler experiences at scale.

By Dineth Jayakody, Pasindu Thenahandi, Sampath Jayarathna

nlp multimodal

arXiv Machine Learning

Jun 2

Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies

arXiv:2508. 20072v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models adapt large vision-language backbones to map images and instructions into robot actions.

By Zhixuan Liang, Yizhuo Li, Tianshuo Yang, Chengyue Wu, Sitong Mao, Liuao Pei, Tian Nian, Shunbo Zhou, Xiaokang Yang, Jiangmiao Pang, Yao Mu, Ping Luo

llms diffusion robotics multimodal

Get the daily briefing