Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv Machine Learning

Jun 2

DeepLatent: Think with Images via Parallel Latent Visual Reasoning

arXiv:2606. 00562v1 Announce Type: cross Abstract: The emerging paradigm of "thinking with images" embeds visual states into intermediate reasoning steps, defining a new frontier for Vision-Language Models.

By Dongchen Lu, Zhimo Li, Mao Shu, Huo Cao

llms rag reinforcement-learning robotics efficiency multimodal benchmarks

arXiv AI

Jun 2

MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction

arXiv:2604. 10169v2 Announce Type: replace Abstract: Trajectory prediction is a key component of autonomous driving systems because future motions directly affect collision checking, behavior planning, and control.

By Wenchang Duan, Zhenguo Gao, Jinguo Xian, Yi Shi

agents robotics efficiency multimodal safety

arXiv Machine Learning

Jun 2

IstGPT: LLM-based Anomaly Detection for Spatial-Temporal Graph in Industrial Systems

arXiv:2606. 01691v1 Announce Type: cross Abstract: Industrial Internet systems face increasing threats from sophisticated industrial control system (ICS) attacks, resulting in critical safety incidents.

By Yuchen Zhang, Ning Xi, Pengbin Feng, Shigang Liu, Jianfeng Ma, Yulong Shen, Yanan Sun, Xiaolin Zhou

llms robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

arXiv:2510. 13774v2 Announce Type: replace Abstract: Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data.

By Dominik J. M\"uhlematter, Lin Che, Ye Hong, Martin Raubal, Nina Wiedemann

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Contrastive Representation Regularization for Vision-Language-Action Models

arXiv:2510. 01711v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have shown strong capabilities in robot manipulation by leveraging rich representations from pre-trained Vision-Language Models (VLMs).

By Taeyoung Kim, Jimin Lee, Myungkyu Koo, Dongyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin

llms robotics multimodal benchmarks

arXiv AI

Jun 2

MOSS-Audio Technical Report

arXiv:2606. 01802v1 Announce Type: cross Abstract: MOSS-Audio is a unified audio-language model for speech, environmental sound, and music understanding, supporting audio captioning, time-aware question answering, timestamped transcription, and audio-grounded reasoning.

By Chen Yang, Chufan Yu, Hanfu Chen, Jie Zhu, Jingqi Chen, Ke Chen, Wenxuan Wang, Yang Wang, Yaozhou Jiang, Yi Jiang, Zhengyuan Lin, Ziqi Chen, Zhaoye Fei, Chenghao Liu, Jun Zhan, Kang Yu, Kexin Huang, Mingshu Chen, Qinyuan Cheng, Ruixiao Li, Shimin Li, Songlin Wang, Yang Gao, Yiyang Zhang, Xipeng Qiu

llms agents nlp fine-tuning multimodal

arXiv Machine Learning

Jun 2

Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies

arXiv:2606. 01151v1 Announce Type: new Abstract: Behavior cloning with high-capacity generative policies achieves strong imitation performance, but is often limited by demonstration coverage and distribution shift.

By Hikmet Simsir, Ozgur S. Oguz

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

arXiv:2606. 01703v1 Announce Type: cross Abstract: We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions.

By Jiashuo Yu, Yao Yao, Boyu Chen, Alex Wang

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

PATRA: Pattern-Aware Alignment and Balanced Reasoning for Time Series Question Answering

arXiv:2602. 23161v4 Announce Type: replace Abstract: Time series reasoning demands both the perception of complex dynamics and logical depth.

By Junkai Lu, Peng Chen, Xingjian Wu, Yang Shu, Chenjuan Guo, Christian S. Jensen, Bin Yang

llms nlp multimodal safety

arXiv Machine Learning

Jun 2

Towards Sparse Video Understanding and Reasoning

arXiv:2602. 13602v2 Announce Type: replace-cross Abstract: We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA).

By Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

llms agents nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

arXiv:2605. 23500v2 Announce Type: replace-cross Abstract: Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis.

By Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms agents computer-vision reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 2

Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning

arXiv:2606. 00851v1 Announce Type: cross Abstract: Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues.

By Sukru Samet Dindar, Riki Shimizu, Xilin Jiang, Nima Mesgarani

multimodal safety

arXiv AI

Jun 2

MURMUR: An Efficient Inference System for Long-Form ASR

arXiv:2606. 01483v1 Announce Type: cross Abstract: Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two.

By Wei-Tzu Lee, Keisuke Kamahori, Baris Kasikci

efficiency multimodal

arXiv AI

Jun 2

An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

arXiv:2606. 00987v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temporal visual reasoning remains underexplored.

By Bingyu Li, Da Zhang, Tao Huo, Zhiyuan Zhao, Junyu Gao, Xuelong Li

llms agents computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 2

STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models

arXiv:2606. 01790v1 Announce Type: cross Abstract: Vision-language-model-based graphical user interface (GUI) agents have shown broad automation capabilities, yet deployment is bottlenecked by a key-value (KV) cache that grows linearly with interaction steps.

By Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 2

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

arXiv:2603. 00171v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details.

By Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li, Man Zhou, Chengjun Xie, Haoxuan Che, Xuanhua He, Jie Zhang

llms multimodal benchmarks

arXiv AI

Jun 2

Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention

arXiv:2512. 10414v2 Announce Type: replace Abstract: Recently, reinforcement learning (RL) has become a common choice in enhancing the reasoning capabilities of vision-language models (VLMs).

By Yang Yu, Zhuangzhuang Chen, Lanqing Li, Xiaomeng Li

llms reinforcement-learning fine-tuning multimodal safety

arXiv AI

Jun 2

UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment

arXiv:2606. 00170v1 Announce Type: cross Abstract: In recent years, emotion recognition based on physiological signals such as electroencephalogram (EEG) has gained considerable attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions.

By Zheng Wang, Shuo Wang, Junhong Wang

llms fine-tuning efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs

arXiv:2606. 00813v1 Announce Type: cross Abstract: Safety alignment in LLMs does not improve monotonically across model generations.

By Subhadip Mitra

llms multimodal benchmarks safety

arXiv AI

Jun 2

Deep Interest Mining for Intent-Enriched Semantic IDs in Multimodal Generative Recommendation

arXiv:2604. 20861v3 Announce Type: replace-cross Abstract: Semantic IDs (SIDs) provide the discrete item vocabulary used by generative recommendation, but their quality depends on what item evidence is preserved before quantization.

By Yangchen Zeng, Jinze Wang

llms rag reinforcement-learning efficiency multimodal

Get the daily briefing