Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,069 stories · RSS feed

arXiv AI

Jun 29

Home3D 1.0: A High-Fidelity Image-to-3D Asset Generation System for Interior Design

arXiv:2606. 27923v1 Announce Type: cross Abstract: We present Home3D 1.

By Yiyun Fei, Guoqiu Li, Jin Song, Chuqiao Wu, Delong Wu, Hong Wu, Ziru Zeng, Haohui Chen, YinDong Kong, Jing Li, Qi Wu, Feng Zhang

computer-vision multimodal

arXiv AI

Jun 29

HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

arXiv:2606. 27627v1 Announce Type: cross Abstract: Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs).

By Artem Ploujnikov, Francesco Verdini, Samir Sadok, Mirco Ravanelli

llms multimodal

arXiv AI

Jun 29

NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning

arXiv:2606. 27826v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly deployed as embodied planners in egocentric environments, where task success requires not only achieving instructed goals but also acting in socially appropriate ways.

By Shiyun Zhao, Xinwei Song, Tianyu Guo, Xiaomeng Gao, Mingyuan Liu, Xu Han, Yuanyuan Zhang, Zhenliang Zhang, Xue Feng, Bo Dai

llms agents robotics multimodal benchmarks

arXiv AI

Jun 29

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

arXiv:2601. 22709v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.

By Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 29

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

arXiv:2606. 27886v1 Announce Type: new Abstract: Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts.

By Ahmed Mohamady, Robin Burchard, Kristof Van Laerhoven

multimodal benchmarks

arXiv AI

Jun 29

Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understanding

arXiv:2606. 27922v1 Announce Type: cross Abstract: Current multimodal reflection mechanisms for long video understanding predominantly rely on closed-loop self-reflection within internal parameters.

By Shuimu Chen, Yuteng Chen, Yuanshen Guan, Zebang Cheng, Zeyu Zhang, Shengqian Qin, Bin Xia, Jiaran Li, Wenming Yang, Fei Ma

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 29

Verifiable Geometry Problem Solving: Solver-Driven Autoformalization and Theorem Proposing

arXiv:2606. 27926v1 Announce Type: new Abstract: Geometry Problem Solving have increasingly adopt the neuro-symbolic paradigm, combining neural intuition with symbolic rigor.

By Can Li, Ting Zhang, Junbo Zhao, Hua Huang

agents reinforcement-learning multimodal

arXiv AI

Jun 29

Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior

arXiv:2604. 03401v4 Announce Type: replace-cross Abstract: Understanding student engagement usually requires time-consuming manual observation or invasive recording that raises privacy concerns.

By Nolan Platt, Sehrish Nizamani, Alp Tural, Elif Tural, Saad Nizamani, Andrew Katz, Yoonje Lee, Nada Basit

llms multimodal safety

arXiv AI

Jun 29

GRAFT: Biological Graph and Hypergraph Benchmarks for Linked Gene Expression and Phenotypic Trait Prediction in Arabidopsis thaliana

arXiv:2606. 27413v1 Announce Type: cross Abstract: Understanding which genes control which traits in an organism remains one of the central challenges in biology.

By Manuel Serna-Aguilera, Vanshika Jindal, Fiona L. Goggin, Jiamei Li, Aranyak Goswami, Alexander Bucksch, Suxing Liu, Khoa Luu

multimodal benchmarks

arXiv AI

Jun 29

S$^2$-VLA: State-Space Guided Vision-Language-Action Models for Long-Horizon Manipulation

arXiv:2606. 27872v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation, but their performance degrades significantly in long-horizon tasks due to cumulative error propagation.

By Zhipeng Xie, Zongyi Han, Xiangyi Wei, Shiliang Sun, Yang Li, Jing Zhao

robotics multimodal benchmarks

arXiv AI

Jun 29

SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion

arXiv:2606. 27876v1 Announce Type: cross Abstract: Spatial intelligence is essential for low-altitude unmanned aerial vehicle (UAV) perception, collaboration, and navigation.

By Haoyu Zhang, Meng Liu, Qianlong Xiang, Kun Wang, Yaowei Wang, Liqiang Nie

llms multimodal benchmarks

arXiv AI

Jun 29

MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy

arXiv:2606. 27652v1 Announce Type: new Abstract: We find that explicit reasoning does not necessarily translate into better multimodal emotion recognition (MER) accuracy, even though it makes predictions more interpretable.

By Zhiyuan Han, Beier Zhu, Wenwen Tong, Chengwei Qin, Xinyi Wang, Jiayu Zhang, Jiangnan Chen, Hewei Guo, Dongchuan Ran, Lewei Lu, Xun Yang

reinforcement-learning multimodal benchmarks

arXiv AI

Jun 29

Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration

arXiv:2606. 28274v1 Announce Type: cross Abstract: Accurate network traffic prediction is a critical element for efficient resource allocation in dynamic urban cellular networks.

By Abdolazim Rezaei, Mehdi Sookhak, Mahboobeh Haghparast

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 29

DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection

arXiv:2606. 27499v1 Announce Type: cross Abstract: Research on agent memory has matured rapidly, but almost entirely on the text side: few existing benchmarks ask, in an interactive environment, when an agent genuinely needs to remember what it saw rather than what it could write down.

By Yujin Tang, Chenming Shang, Ruize Xu, Nikhil Singh

llms agents multimodal benchmarks safety

arXiv AI

Jun 29

ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

arXiv:2606. 27974v1 Announce Type: cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge.

By ZhengXian Wu, Hangrui Xu, Kai Shi, Zhuohong Chen, Yunyao Yu, Chuanrui Zhang, Zirui Liao, Jun Yang, Zhenyu Yang, Haonan Lu, Haoqian Wang

rag agents nlp reinforcement-learning fine-tuning multimodal

arXiv AI

Jun 29

RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance

arXiv:2606. 27766v1 Announce Type: cross Abstract: Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe.

By Shiqiang Gong

diffusion reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jun 29

JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications

arXiv:2606. 28070v1 Announce Type: new Abstract: JD.

By Oxygen AIIC, Chan Long, Chao Liu, Chaofan Chen, Chaohui Dong, Chunyuan Guo, Danping Liu, Debin Liu, Deping Xiang, Fulai Xu, Guangyue Liu, Hao Li, Huichun Hu, Jian Yang, Jianan Wang, Jianbo Zhao, Jiaoyang Li, Jiaxing Wang, Jinglong Li, Jinjin Guo, Jun Fang, Jun Liu, Kai Zhou, Li Wang, Lili Gao, Liying Chen, Luning Yang, Mengdi Zhou, Pengzhang Liu, Qi Lv, Qianyun Wang, Qixia Jiang, Ruyue Li, Shimu Liang, Shuxing Wang, Sijie Zhang, Siqi Li, Tianhao Gao, Wang Ke, Weihu Huang, Wencan Lai, Wenjie Zhang, Xiaohui Zhang, Xiaojing Dong, Ya Liu, Yifeng Zhang, Yixiang Wang, Yongtai Zhang, Yongyi Liao, Zhaoru Chen, Zhen Chen, Zhiyong Ma, Zhiyuan Liu, Zhongwei Liu, Ziyan Xing

llms rag efficiency multimodal

arXiv AI

Jun 29

Benchmarking Multi-Modal Graph-based Social Media Popularity Prediction

arXiv:2606. 27539v1 Announce Type: cross Abstract: Social media popularity prediction aims to forecast the future reach or influence of online content from early-stage observations.

By Utkarsh Sahu, Zhisheng Qi, Li Zhu, Yizhao Yang, Jun Li, Ryan Rossi, Yu Wang

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 29

Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings

arXiv:2606. 27543v1 Announce Type: cross Abstract: The variations in vocal effort range (e.

By Zahra Omidi, John H. L. Hansen

multimodal benchmarks

arXiv AI

Jun 29

GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models

arXiv:2601. 18197v2 Announce Type: replace Abstract: While Large Vision-Language Models (LVLMs) have significantly advanced GUI agents' capabilities in parsing textual instructions, interpreting screen content, and executing tasks, a critical challenge persists: the irreversibility of agent operations-where a single erroneous action can trigger catastrophic deviations.

By Shaokang Wang, Pei Fu, Ruoceng Zhang, Shaojie Zhang, Xiuwen Xi, Jiahui Yang, Bin Qin, Ying Huang, Zhenbo Luo, Jian Luan

llms agents multimodal

Get the daily briefing