Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,406 stories · RSS feed

arXiv AI

Jun 6

UniVoice: A Unified Model for Speech and Singing Voice Generation

arXiv:2606. 05852v1 Announce Type: cross Abstract: Text-to-speech (TTS) and singing voice synthesis (SVS) both aim to generate human vocal audio from symbolic inputs, but they impose different requirements on the generation process.

By Junjie Zheng, Huixin Xue, Shihong Ren, Chaofan Ding, Hao Liu, Zihao Chen

llms diffusion multimodal safety

arXiv AI

Jun 6

Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation

arXiv:2606. 06076v1 Announce Type: new Abstract: While vision-language models excel at general multimodal understanding, they still struggle with visual spatial planning.

By Haocheng Luo, Jiahui Liu, Ruicheng Zhang, Zhizhou Zhong, Jiaqi Huang, Zunnan Xu, Quan Shi, Jun Zhou, Xiu Li

llms efficiency multimodal benchmarks

arXiv AI

Jun 6

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

arXiv:2606. 05177v1 Announce Type: cross Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text.

By Manh Luong, Tamas Abraham, Junae Kim, Amar Kaur, Rollin Omari, Gholamreza Haffari, Trang Vu, Lizhen Qu, Dinh Phung

llms multimodal benchmarks safety

arXiv AI

Jun 6

HomeWorld: A Unified Floorplan-to-Furnished Framework for Generating Controllable, Densely Interactive Whole-Home Scenes

arXiv:2606. 06390v1 Announce Type: cross Abstract: Indoor scene generation is crucial for robot simulation and modern interior design.

By Wenbo Li, Xiaoliang Ju, Zipeng Qin, Rongyao Fang, Hongsheng Li

llms diffusion robotics multimodal

arXiv AI

Jun 6

MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA

arXiv:2606. 05749v1 Announce Type: cross Abstract: Iterative retrieval-reasoning agents have recently shown promise for multimodal long-document question answering.

By Kaifeng Chen, Hongtao Liu, Qiyao Peng, Jian Yang, Yongqiang Liu, Xiaochen Zhang, Qing Yang

agents nlp multimodal

arXiv AI

Jun 6

I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisition

arXiv:2606. 05316v1 Announce Type: new Abstract: Multimodal memes are dynamic and often require up to date background knowledge for interpretation.

By Shanhong Liu, Rui Cao, Pai Chet Ng, De Wen Soh

multimodal benchmarks

arXiv AI

Jun 6

Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs

arXiv:2606. 05966v1 Announce Type: cross Abstract: Understanding and reasoning about the physical world is the foundation of intelligent behavior, yet state-of-the-art vision-language models (VLMs) still fail at causal physical reasoning, often producing plausible but incorrect answers.

By Tianyi Tang, Zhuoyi Lin, Zeyu Feng, Tianyi Ma, Yew-Soon Ong, Ivor Tsang, Haiyan Yin

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

MAVEN A Multi-Agent Framework for Multicultural Text-to-Video Generation

arXiv:2605. 16716v4 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv AI

Jun 6

HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling

arXiv:2510. 00054v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding tasks.

By Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, Bo Zheng

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 6

MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

arXiv:2605. 28579v2 Announce Type: replace Abstract: Large language models (LLMs) have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design.

By Xiaoyu Dong, Zhi Li, Xiao-Ming Wu

llms multimodal benchmarks safety

arXiv AI

Jun 6

Benchmark Everything Everywhere All at Once

arXiv:2606. 06462v1 Announce Type: new Abstract: Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance.

By Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai, Bokang Yang, Wencheng Han, Xiao-Hui Li, Xiangyu Yue

llms agents multimodal benchmarks

arXiv AI

Jun 6

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

arXiv:2606. 05979v1 Announce Type: cross Abstract: We propose world-language-action (WLA) models as a new class of embodied foundation models.

By Yi Yang, Zhihong Liu, Siqi Kou, Yiyang Chen, Yanzhe Hu, Jianbo Zhou, Boyuan Zhao, Zhijie Wei, Xiao Xia, Xueqi Li, Pengfei Liu, Zhijie Deng

llms diffusion robotics multimodal benchmarks

arXiv AI

Jun 6

MAviS: A Multimodal Conversational Assistant For Avian Species

arXiv:2603. 07294v2 Announce Type: replace-cross Abstract: Fine-grained understanding and species-specific multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring.

By Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jinxing Zhou, Fahad Shabzan Khan, Rao Anwer, Salman Khan, Hisham Cholakkal

llms nlp multimodal benchmarks

arXiv AI

Jun 6

InfoShield: Privacy-Preserving Speech Representations for Mental Health Screening via Information-Theoretic Optimization

arXiv:2606. 05561v1 Announce Type: cross Abstract: Speech-based mental health screening offers scalable depression detection, yet clinical deployment faces a significant barrier: users' privacy concerns about demographic information exposure.

By Xueyang Wu, Siyuan Liu, Kezhuo Yang, Guang Ling

rag multimodal benchmarks safety

arXiv AI

Jun 6

TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models

arXiv:2606. 06285v1 Announce Type: new Abstract: Time series foundation models (TS-FMs) aim to learn generalizable temporal representations that can be adapted to a wide range of downstream tasks.

By Ziwen Kan, Yishuo Chen, Kecheng Li, Andrew Wen, Xiaomeng Wang, Liwei Wang, Jihao Duan, Song Wang, Hongfang Liu, Tianlong Chen

nlp multimodal benchmarks

arXiv AI

Jun 6

DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments

arXiv:2606. 06217v1 Announce Type: cross Abstract: When a disaster unfolds, responders must answer not only what is happening, but also why it is happening, what will happen next, and what to do now, often from noisy low-altitude UAV views and under tight on-site compute constraints.

By Tan Zhang, Quanyou Li, Lu Zhang, Jun Liu, Xiaofeng Zhu, Ping Hu

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

FUSAR-GPT : A Spatiotemporal Feature-Embedded and Two-Stage Decoupled Visual Language Model for SAR Imagery

arXiv:2602. 19190v4 Announce Type: replace-cross Abstract: Research on the intelligent interpretation of all-weather, all-time Synthetic Aperture Radar (SAR) is crucial for advancing remote sensing applications.

By Xiaokun Zhang, Yi Yang, Ziqi Ye, Baiyun, Xiaorong Guo, Qingchen Fang, Ruyi Zhang, Xinpeng Zhou, Haipeng Wang

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jun 6

CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving

arXiv:2606. 06219v1 Announce Type: cross Abstract: End-to-end autonomous driving models often struggle to balance multi-modal maneuver generation with real-time inference constraints.

By Yining Xing, Zehong Ke, Zhiyuan Liu, Yanbo Jiang, Wenhao Yu, Jianqiang Wang

agents diffusion robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

Edit-R2: Context-Aware Reinforcement Learning for Multi-Turn Image Editing

arXiv:2606. 05950v1 Announce Type: new Abstract: Text-guided image editing has advanced rapidly with diffusion models and unified multimodal foundation models.

By Yuxiao Ye, Haoran He, Fangyuan Kong, Xintao Wang, Pengfei Wan, Kun Gai, Ling Pan

diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jun 6

Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation

arXiv:2606. 05290v1 Announce Type: cross Abstract: Recent progress in generative modeling has made safety control a central challenge, yet existing approaches remain largely model-specific, requiring retraining or tailored interventions for each new architecture.

By Tobia Poppi, Silvia Cappelletti, Sara Sarto, Florian Schiffers, Garin Kessler, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

llms diffusion multimodal safety

Get the daily briefing