Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,396 stories · RSS feed

arXiv AI

Jul 2

Learning When to Listen: Gated Affect Fusion for Human Motion Prediction

arXiv:2607. 00296v1 Announce Type: cross Abstract: Human motion forecasting in unconstrained real-world videos remains challenging due to the ambiguity of future behaviors and the presence of noisy multimodal observations.

By Jingni Huang

llms multimodal

arXiv Machine Learning

Jul 2

VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

arXiv:2511. 17731v2 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) prompting has proven remarkably effective for eliciting complex reasoning in large language models (LLMs).

By Lingxiao Li, Yifan Wang, Xinyan Gao, Chen Tang, Xiangyu Yue, Chenyu You

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 2

Towards Developing a Multimodal Chat Assistant for University Stakeholders: RAG-based Approach

arXiv:2607. 01115v1 Announce Type: cross Abstract: University stakeholders often face difficulties in accessing timely and reliable information, especially in developing countries, where there are very few intelligent support systems.

By Md Abu Hanif Shaikh, Abdullah Al Shafi

llms rag efficiency multimodal safety

arXiv Machine Learning

Jul 2

AdaBoosting Text Prompts for Vision-Language Models

arXiv:2607. 00684v1 Announce Type: new Abstract: The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts.

By Seokhee Jin, Changhwan Sung, Sunung Mun, Hoyoung Kim, Jungseul Ok

llms multimodal benchmarks

arXiv Machine Learning

Jul 2

IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

arXiv:2603. 16432v3 Announce Type: replace-cross Abstract: Unsupervised physical parameter estimation from video lacks a common benchmark: existing methods evaluate on non-overlapping synthetic data, the sole real-world dataset is restricted to single-body systems, and no established protocol addresses governing-equation identification.

By Rasul Khanbayov, Mohamed Rayan Barhdadi, Erchin Serpedin, Hasan Kurban

llms multimodal benchmarks

arXiv Machine Learning

Jul 2

Information-Regularized Attention for Visual-Centric Reasoning

arXiv:2607. 00434v1 Announce Type: cross Abstract: Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning.

By Guohao Sun, Xiaofang Wang, Yash Patel, Mengchen Liu, Zhiqiang Tao, Praveen Krishnan

llms rag fine-tuning multimodal safety

arXiv Machine Learning

Jul 2

Rosetta: Composable Native Multimodal Pretraining

arXiv:2607. 00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge.

By Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Ping Tan

llms diffusion multimodal

arXiv Machine Learning

Jul 2

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment

arXiv:2607. 00410v1 Announce Type: cross Abstract: Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals.

By Zhenhang Li, Xin Zhou, Hao Deng, Lijun Yin

rag diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 2

Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility

arXiv:2607. 00228v1 Announce Type: cross Abstract: Modern time-domain surveys such as the Zwicky Transient Facility (ZTF) generate hundreds of thousands of alerts each night, making real-time decisions for follow-up observations a central challenge in time-domain astronomy.

By Ved G. Shah, Nabeel Rehemtulla, Adam A. Miller, Sushant Sharma Chaudhary, Michael W. Coughlin, Antoine Le Calloch, Matthew J. Graham, Joahan Castaneda Jaimes, Theophile Jegou du Laz, Ashish A. Mahabal, Frank J. Masci, Josiah Purdum, Reed Riddle, Jesper Sollerman, Anastasia Wei, Mansi M. Kasliwal

efficiency multimodal benchmarks

arXiv Machine Learning

Jul 2

Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

arXiv:2505. 19614v2 Announce Type: replace Abstract: Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities.

By Sanghyuk Chun, Olga Russakovsky

multimodal benchmarks safety

arXiv AI

Jul 2

Interact3D: Compositional 3D Generation of Interactive Objects

arXiv:2603. 16085v2 Announce Type: replace-cross Abstract: Recent breakthroughs in 3D generation have enabled the synthesis of high-fidelity individual assets.

By Hui Shan, Keyang Luo, Ming Li, Sizhe Zheng, Yanwei Fu, Zhen Chen, Xiangru Huang

llms agents multimodal safety

arXiv AI

Jul 2

From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping

arXiv:2607. 00530v1 Announce Type: cross Abstract: Improvements in the technical performance of human--robot interaction (HRI) systems do not automatically translate into differences that human users can detect during live interaction.

By Jian Song, Tian Zi, Shen Guanting

llms computer-vision robotics multimodal benchmarks

arXiv AI

Jul 2

SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

arXiv:2603. 16859v2 Announce Type: replace Abstract: Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text.

By Tianyu Xie, Jinfa Huang, Yuexiao Ma, Rongfang Luo, Yan Yang, Wang Chen, Yuhui Zeng, Yixuan Zou, Qingchuan Ma, Zhiqiang Lu, Ruize Fang, Xiawu Zheng, Jiebo Luo, Rongrong Ji

llms multimodal benchmarks

arXiv AI

Jul 2

SleepLM: Natural-Language Intelligence for Human Sleep

arXiv:2602. 23605v2 Announce Type: replace Abstract: We present SleepLM, a family of sleep-language foundation models that enable human sleep alignment, interpretation, and interaction with natural language.

By Zongzhe Xu, Zitao Shuai, Eideen Mozaffari, Ravi S. Aysola, Rajesh Kumar, Yuzhe Yang

multimodal benchmarks safety

arXiv AI

Jul 2

Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

arXiv:2607. 00514v1 Announce Type: cross Abstract: Automatic understanding of dynamic 4D point clouds, the 3D-point sequences captured over time by depth sensors and LiDAR, is central to robotics and embodied perception.

By Trung Thanh Nguyen, Hai Nguyen-Truong, Tu Vo, Hoang M. Truong, Tuan-Anh Vu

rag diffusion robotics fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 2

MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

arXiv:2607. 00491v1 Announce Type: cross Abstract: Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input.

By Leyuan Yu, Xiao Tang, Minghao Liu, Xinyuan Li, Xiaokai Bai, Sheng Zhou, Qunshu Lin, Weihao Xuan, Naoto Yokoya

llms multimodal benchmarks

arXiv AI

Jul 2

What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models

arXiv:2607. 00283v1 Announce Type: cross Abstract: Autonomous vehicles must safely navigate complex environments where planning-critical agents may be hidden from view.

By Amirhosein Chahe, Tyler Naes, Jovin D'sa, Faizan M. Tariq, Sangjae Bae, Lifeng Zhou, David Isele

llms agents robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 2

ASPIRE: Agentic /Skills Discovery for Robotics

arXiv:2607. 00272v1 Announce Type: cross Abstract: Traditional robot programming is challenging: it requires orchestrating multimodal perception, managing physical contact dynamics, and handling diverse configurations and execution failures.

By Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi "Jim" Fan, Guanzhi Wang

agents robotics multimodal

arXiv AI

Jul 2

UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios

arXiv:2511. 18050v1 Announce Type: cross Abstract: Diffusion transformers have recently delivered strong text-to-image generation around 1K resolution, but we show that extending them to native 4K across diverse aspect ratios exposes a tightly coupled failure mode spanning positional encoding, VAE compression, and optimization.

By Tian Ye, Song Fei, Lei Zhu

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 2

Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts

arXiv:2607. 00666v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models often fail to perform the same learned tasks under environmental shifts, such as changes in camera pose and shifts to a different but similar robot (e.

By Taewook Kang, Taeheon Kim, Donghyun Shin, Jonghyun Choi

robotics multimodal safety

Get the daily briefing