Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,406 stories · RSS feed

arXiv AI

Jun 6

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

arXiv:2606. 05677v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs.

By Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun, Yuanteng Chen, Tao Liu, Lan Yang, Longteng Guo, Honggang Zhang

llms agents robotics multimodal benchmarks

arXiv AI

Jun 6

OneReason Technical Report

arXiv:2606. 06260v1 Announce Type: cross Abstract: Generative recommendation models in the OneRec family have been widely deployed in many real-world services, such as short-video, live-streaming, advertising, and e-commerce.

By OneRec Team, Biao Yang, Boyang Ding, Chenglong Chu, Dunju Zang, Fei Pan, Han Li, Hao Jiang, Honghui Bao, Huanjie Wang, Jian Liang, Jiangxia Cao, Jiao Ou, Jiaxin Deng, Jinghao Zhang, Kun Gai, Lu Ren, Peiru Du, Pengfei Zheng, Rongzhou Zhang, Ruiming Tang, Shiyao Wang, Siyang Mao, Siyuan Lou, Teng Shi, Wei Yuan, Wenlong Xu, Xingchen Liu, Xingmei Wang, Xinqi Jin, Yan Sun, Yan Wang, Yifei Hu, Yingzhi He, Yufei Ye, Yuhao Wang, Yunhao Zhou, Yuqin Dai, Zhao Liu, Zhipeng Wei, Zhixin Ling, Ziming Li, Zixing Zhang, Ziyuan Liu, An Zhang, Changxin Lao, Chaoyi Ma, Chengru Song, Defu Lian, Fan Yang, Guowang Zhang, Hao Peng, Jiayao Shen, Jie Chen, Jun Xu, Junmin Chen, Kun Zhang, Kuo Cai, Mingxing Wen, Minmao Wang, Minxuan Lv, Qi Zhang, Qiang Luo, Sheng Yu, Shijie Li, Shijie Yi, Shuang Yang, Shugui Liu, Shuni Chen, Tinghai Zhang, Tingting Gao, Xiang Wang, Xiangyu Wu, Xiangyu Zhao, Xiao Lv, Xiaoyou Zhou, Xuming Wang, Yong Du, Zejian Zhang, Zhaojie Liu, Zhiyang Zhang, Zhuang Zhuang, Ziqi Wang, Ziyi Zhao

llms reinforcement-learning fine-tuning multimodal

arXiv AI

Jun 6

HomeWorld: A Unified Floorplan-to-Furnished Framework for Generating Controllable, Densely Interactive Whole-Home Scenes

arXiv:2606. 06390v1 Announce Type: cross Abstract: Indoor scene generation is crucial for robot simulation and modern interior design.

By Wenbo Li, Xiaoliang Ju, Zipeng Qin, Rongyao Fang, Hongsheng Li

llms diffusion robotics multimodal

arXiv AI

Jun 6

Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

arXiv:2606. 05445v1 Announce Type: new Abstract: We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks.

By Jiateng Liu, Bingxuan Li, Zhenhailong Wang, Rushi Wang, Kaiwen Hong, Cheng Qian, Jiayu Liu, Denghui Zhang, Katherine Driggs-Campbell, Manling Li, Heng Ji

llms agents computer-vision multimodal benchmarks

arXiv Machine Learning

Jun 5

Drishti AI-Event Guardian: An Intelligent Real-Time Crowd Monitoring and Emergency Response System for Mass Gathering Events

arXiv:2606. 05185v1 Announce Type: cross Abstract: Mass gathering events are associated with critical safety incidents caused by insufficient crowd monitoring and inadequate emergency response coordination.

By Ritabrata Roy Choudhury, Arkajyoti Karmakar, Rudra Pratap Mitra

multimodal safety

arXiv Machine Learning

Jun 5

ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

arXiv:2606. 05718v1 Announce Type: cross Abstract: On-policy distillation (OPD) improves reasoning by training a student on trajectories sampled from its own policy under supervision from a teacher.

By Kanghui Tian, Siyuan Liu, Ziang Yan, Sheng Xia, Shuai Dong, Yi Wang

efficiency multimodal benchmarks

arXiv Machine Learning

Jun 5

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models

arXiv:2606. 05531v1 Announce Type: cross Abstract: Despite the rapid progress of Vision-Language Models (VLMs), the field lacks benchmarks that rigorously diagnose their true reasoning abilities and chart meaningful progress toward human-like multimodal intelligence.

By Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, Mohamed Hefeeda, Ehsaneddin Asgari

llms multimodal benchmarks

arXiv Machine Learning

Jun 5

GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention

arXiv:2606. 06249v1 Announce Type: cross Abstract: Transformer-based multimodal models rely on attention mechanisms to integrate information across heterogeneous modalities.

By Giordano Cicchetti, Eleonora Grassucci, Danilo Comminiello

llms multimodal

arXiv Machine Learning

Jun 5

Steering Vectors are an Adversarial Attack Surface

arXiv:2606. 05958v1 Announce Type: new Abstract: Activation steering has become a popular way to control Large Language Model (LLM) behavior without fine-tuning.

By Abzal Aidakhmetov, Donato Crisostomi, Tommaso Mencattini, Adrian Robert Minut, Iacopo Masi, Emanuele Rodol\`a

llms fine-tuning multimodal safety

arXiv Machine Learning

Jun 5

Learning Self-Correction in Vision-Language Models via Rollout Augmentation

arXiv:2602. 08503v2 Announce Type: replace-cross Abstract: Self-correction is essential for solving complex reasoning problems in vision-language models (VLMs).

By Yi Ding, Ziliang Qiu, Bolian Li, Ruqi Zhang

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 5

To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection

arXiv:2606. 05931v1 Announce Type: cross Abstract: When retrieving a person from a video archive by voice and face, should the system be multimodal or not?

By Erfan Loweimi, Mengjie Qian, Kate Knill, Guanfeng Wu, Chi-Ho Chan, Abbas Haider, Muhammad Awan, Josef Kittler, Hui Wang, Mark Gales

multimodal benchmarks

arXiv Machine Learning

Jun 5

Test-Time Training for Visual Foresight Vision-Language-Action Models

arXiv:2605. 08215v2 Announce Type: replace-cross Abstract: Visual Foresight VLA (VF-VLA) has become a prominent architectural choice in the recent VLA due to its impressive performance.

By Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, Hongseok Kang, Chanyoung Park

multimodal

arXiv Machine Learning

Jun 5

HANDOFF: Humanoid Agentic Task-Space Whole-Body Control via Distilled Complementary Teachers

arXiv:2606. 06493v1 Announce Type: cross Abstract: For a humanoid robot to be deployed in the real world, the choice of command space (i.

By Lizhi Yang, Junheng Li, Nehar Poddar, Yiling Hou, Gio Huh, Robert Griffin, Georgia Gkioxari, Aaron Ames

agents robotics fine-tuning efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 5

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

arXiv:2510. 05544v2 Announce Type: replace-cross Abstract: Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment.

By Ryan Solgi, Parsa Madinei, Jiayi Tian, Rupak Swaminathan, Jing Liu, Nathan Susanj, Zheng Zhang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jun 5

Discrete Causal Representations from Heterogeneous Domains: A Bayesian Approach with Social Survey Applications

arXiv:2606. 06288v1 Announce Type: cross Abstract: Causal representation learning aims to infer the high-level latent causal concepts that give rise to observed low-level measurements.

By Ankur Garg, Michael Stettler, Aaron Schein, Julius von K\"ugelgen

multimodal safety

arXiv Machine Learning

Jun 5

Differentiable Efficient Operator Search

arXiv:2606. 05232v1 Announce Type: new Abstract: Efficient multimodal foundation models often rely on manually designed token-reduction operators, such as pruning, merging, pooling, and adaptive reweighting.

By Xiaohuan Pei, Jiyuan Zhang, Yuanfan Guo, Weiguo Feng, Tao Huang, Cho-Jui Hsieh, Chang Xu

efficiency multimodal benchmarks

arXiv Machine Learning

Jun 5

Harnessing Generalist Agents for Contextualized Time Series

arXiv:2606. 05404v1 Announce Type: cross Abstract: Time series are often embedded in rich contexts that are essential for holistic modeling.

By Zihao Li, Kaifeng Jin, Yuanchen Bei, Jiaru Zou, Avaneesh Kumar, Xuying Ning, Yanjun Zhao, Mengting Ai, Baoyu Jing, Hanghang Tong, Jingrui He

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 5

Equivariant Neural Belief Propagation

arXiv:2606. 06344v1 Announce Type: new Abstract: Probabilistic inference over spatially embedded variables requires beliefs that respect $SE(3)$ symmetry, yet existing equivariant networks produce only scalars and vectors -- not the rank-2 precision tensors needed for anisotropic uncertainty, and single-component messages collapse multi-modal energy landscapes to physically meaningless averages.

By Zehua Cheng, Wei Dai, Jiahao Sun

agents diffusion robotics multimodal

arXiv Machine Learning

Jun 5

Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding

arXiv:2505. 05026v5 Announce Type: replace-cross Abstract: User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept.

By Jaehyun Jeon, Min Soo Kim, Jang Han Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 5

Electricity price forecasting across Norway's five bidding zones in the post-crisis era

arXiv:2604. 26634v2 Announce Type: replace Abstract: Norway's electricity market is heavily dominated by hydropower, but the 2021-2022 energy crisis and stronger integration with Continental Europe have fundamentally altered price formation, reducing the reliability of forecasting models calibrated on historical data.

By My Thi Diem Phan, Trung Tuyen Truong, Hoai Phuong Ha, Dat Thanh Nguyen

multimodal benchmarks safety

Get the daily briefing