Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,406 stories · RSS feed

arXiv AI

Jun 6

Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models

arXiv:2606. 05833v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames.

By Haibo Wang, Lifu Huang

llms diffusion multimodal benchmarks

arXiv AI

Jun 6

Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads

arXiv:2606. 05843v1 Announce Type: cross Abstract: While Multimodal Large Language Models (MLLMs) demonstrate remarkable proficiency on complex vision-language tasks, the mechanisms by which they extract query-relevant visual features from complex, noisy contexts remain opaque.

By Ruoxi Sun, Quantong Qiu, Juntao Li, Zecheng Tang, Yihang Lou, Min Zhang

llms efficiency multimodal safety

arXiv AI

Jun 6

Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming

arXiv:2606. 05233v1 Announce Type: cross Abstract: Recent computer-using-agent (CUA) red-teaming papers report prompt-injection attack success rates (ASR) of 42-98%, but these headline numbers cluster on retired models and on the most-vulnerable model in each paper's panel.

By Nicholas Saban

llms agents reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 6

HomeWorld: A Unified Floorplan-to-Furnished Framework for Generating Controllable, Densely Interactive Whole-Home Scenes

arXiv:2606. 06390v1 Announce Type: cross Abstract: Indoor scene generation is crucial for robot simulation and modern interior design.

By Wenbo Li, Xiaoliang Ju, Zipeng Qin, Rongyao Fang, Hongsheng Li

llms diffusion robotics multimodal

arXiv AI

Jun 6

Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition

arXiv:2606. 05678v1 Announce Type: cross Abstract: Automatic speech recognition (ASR) systems have become widely used for multilingual speech-to-text transcription.

By Yifan Liao, Zongmin Zhang, Zhen Sun, Yuhui Sun, Xinhu Zheng, Xinlei He

multimodal benchmarks safety

arXiv AI

Jun 6

UniVoice: A Unified Model for Speech and Singing Voice Generation

arXiv:2606. 05852v1 Announce Type: cross Abstract: Text-to-speech (TTS) and singing voice synthesis (SVS) both aim to generate human vocal audio from symbolic inputs, but they impose different requirements on the generation process.

By Junjie Zheng, Huixin Xue, Shihong Ren, Chaofan Ding, Hao Liu, Zihao Chen

llms diffusion multimodal safety

arXiv AI

Jun 6

Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

arXiv:2606. 05445v1 Announce Type: new Abstract: We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks.

By Jiateng Liu, Bingxuan Li, Zhenhailong Wang, Rushi Wang, Kaiwen Hong, Cheng Qian, Jiayu Liu, Denghui Zhang, Katherine Driggs-Campbell, Manling Li, Heng Ji

llms agents computer-vision multimodal benchmarks

arXiv AI

Jun 6

InfoShield: Privacy-Preserving Speech Representations for Mental Health Screening via Information-Theoretic Optimization

arXiv:2606. 05561v1 Announce Type: cross Abstract: Speech-based mental health screening offers scalable depression detection, yet clinical deployment faces a significant barrier: users' privacy concerns about demographic information exposure.

By Xueyang Wu, Siyuan Liu, Kezhuo Yang, Guang Ling

rag multimodal benchmarks safety

arXiv AI

Jun 6

OneReason Technical Report

arXiv:2606. 06260v1 Announce Type: cross Abstract: Generative recommendation models in the OneRec family have been widely deployed in many real-world services, such as short-video, live-streaming, advertising, and e-commerce.

By OneRec Team, Biao Yang, Boyang Ding, Chenglong Chu, Dunju Zang, Fei Pan, Han Li, Hao Jiang, Honghui Bao, Huanjie Wang, Jian Liang, Jiangxia Cao, Jiao Ou, Jiaxin Deng, Jinghao Zhang, Kun Gai, Lu Ren, Peiru Du, Pengfei Zheng, Rongzhou Zhang, Ruiming Tang, Shiyao Wang, Siyang Mao, Siyuan Lou, Teng Shi, Wei Yuan, Wenlong Xu, Xingchen Liu, Xingmei Wang, Xinqi Jin, Yan Sun, Yan Wang, Yifei Hu, Yingzhi He, Yufei Ye, Yuhao Wang, Yunhao Zhou, Yuqin Dai, Zhao Liu, Zhipeng Wei, Zhixin Ling, Ziming Li, Zixing Zhang, Ziyuan Liu, An Zhang, Changxin Lao, Chaoyi Ma, Chengru Song, Defu Lian, Fan Yang, Guowang Zhang, Hao Peng, Jiayao Shen, Jie Chen, Jun Xu, Junmin Chen, Kun Zhang, Kuo Cai, Mingxing Wen, Minmao Wang, Minxuan Lv, Qi Zhang, Qiang Luo, Sheng Yu, Shijie Li, Shijie Yi, Shuang Yang, Shugui Liu, Shuni Chen, Tinghai Zhang, Tingting Gao, Xiang Wang, Xiangyu Wu, Xiangyu Zhao, Xiao Lv, Xiaoyou Zhou, Xuming Wang, Yong Du, Zejian Zhang, Zhaojie Liu, Zhiyang Zhang, Zhuang Zhuang, Ziqi Wang, Ziyi Zhao

llms reinforcement-learning fine-tuning multimodal

arXiv AI

Jun 6

AgenticRL: Self-Refining Agentic Reinforcement Learning for Vision-Conditioned UAV Navigation

arXiv:2606. 03963v2 Announce Type: replace-cross Abstract: Deep reinforcement learning has shown strong potential for enabling autonomous robots to learn complex navigational tasks.

By Roohan Ahmed Khan, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Dzmitry Tsetserukou

llms agents reinforcement-learning robotics fine-tuning multimodal

arXiv AI

Jun 6

TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies

arXiv:2606. 06491v1 Announce Type: cross Abstract: Robot manipulation alternates between low-risk transit phases that call for fast execution and high-risk contact stages that demand slow, precise motion.

By Dong Jing, Jingchen Nie, Tianqi Zhang, Jiaqi Liu, Huaxiu Yao, Zhiwu Lu, Mingyu Ding

reinforcement-learning robotics efficiency multimodal

arXiv AI

Jun 6

Using street view images and visual LLMs to predict heritage values for governance support: Risks, ethics, and policy implications

arXiv:2601. 06056v2 Announce Type: replace-cross Abstract: During 2025 and 2026, the Energy Performance of Buildings Directive is being implemented in the European Union member states, requiring all member states to have National Building Renovation Plans.

By Tim Johansson, Mikael Mangold, Kristina Dabrock, Anna Donarelli, Ingrid Campo-Ruiz

llms multimodal

arXiv AI

Jun 6

DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments

arXiv:2606. 06217v1 Announce Type: cross Abstract: When a disaster unfolds, responders must answer not only what is happening, but also why it is happening, what will happen next, and what to do now, often from noisy low-altitude UAV views and under tight on-site compute constraints.

By Tan Zhang, Quanyou Li, Lu Zhang, Jun Liu, Xiaofeng Zhu, Ping Hu

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

UNIVID: Unified Vision-Language Model for Video Moderation

arXiv:2606. 05748v1 Announce Type: cross Abstract: Global-scale video moderation faces a dual challenge: the need for fine-grained multi-modal reasoning and the demand for interpretable outputs to support downstream enforcement.

By Kejuan Yang, Yizhuo Zhang, Mingyuan Du, Yue Zhang, Dixin Zheng, Kaili Zhao, Yang Xiao, Hanzhong Liang, Kenan Xiao

llms multimodal safety

arXiv AI

Jun 6

Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs

arXiv:2606. 05966v1 Announce Type: cross Abstract: Understanding and reasoning about the physical world is the foundation of intelligent behavior, yet state-of-the-art vision-language models (VLMs) still fail at causal physical reasoning, often producing plausible but incorrect answers.

By Tianyi Tang, Zhuoyi Lin, Zeyu Feng, Tianyi Ma, Yew-Soon Ong, Ivor Tsang, Haiyan Yin

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

arXiv:2606. 05979v1 Announce Type: cross Abstract: We propose world-language-action (WLA) models as a new class of embodied foundation models.

By Yi Yang, Zhihong Liu, Siqi Kou, Yiyang Chen, Yanzhe Hu, Jianbo Zhou, Boyuan Zhao, Zhijie Wei, Xiao Xia, Xueqi Li, Pengfei Liu, Zhijie Deng

llms diffusion robotics multimodal benchmarks

arXiv AI

Jun 6

MPCoT: Reward-Guided Multi-Path Latent Reasoning for Test-Time Scalable Vision-Language-Action

arXiv:2606. 06245v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies remain brittle in long-horizon and high-uncertainty control, where one-pass action decoding provides limited inference-time deliberation.

By Boyang Zhang, Lianlei Shan

multimodal

arXiv AI

Jun 6

I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisition

arXiv:2606. 05316v1 Announce Type: new Abstract: Multimodal memes are dynamic and often require up to date background knowledge for interpretation.

By Shanhong Liu, Rui Cao, Pai Chet Ng, De Wen Soh

multimodal benchmarks

arXiv AI

Jun 6

Assessing the Geographic Diversity of AI's Platial Representations in Image Generation

arXiv:2606. 05188v1 Announce Type: cross Abstract: (Gen)AI diversity is not merely an ethical issue.

By Zilong Liu, Krzysztof Janowicz, Mina Karimi

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 6

Edit-R2: Context-Aware Reinforcement Learning for Multi-Turn Image Editing

arXiv:2606. 05950v1 Announce Type: new Abstract: Text-guided image editing has advanced rapidly with diffusion models and unified multimodal foundation models.

By Yuxiao Ye, Haoran He, Fangyuan Kong, Xintao Wang, Pengfei Wan, Kun Gai, Ling Pan

diffusion reinforcement-learning multimodal benchmarks

Get the daily briefing