Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,393 stories · RSS feed

arXiv AI

Jun 16

Toward Vibe Medicine: A Self-Evolving Multi-Agent Framework for Clinical Decision Support

arXiv:2606. 15504v1 Announce Type: new Abstract: In recent years, the advances of large language models and autonomous agents have revolutionized the healthcare field, facilitating diagnosis and improving treatment results.

By Qianxue Zhang, Yiming Ren, Shihuan Qin, Xiao Zhang, Liao Zhang, Jinyang Huang, Zhengliang Liu, Chenbin Liu, Hongying Feng, Jingyuan Chen, Yuzhen Ding, Weihang You, Hanqi Jiang, Yi Pan, Yifan Zhou, Junhao Chen, Lifeng Chen, Wei Liu, Tianming Liu, Zengren Zhao, Lian Zhang

llms agents multimodal safety

arXiv AI

Jun 16

What Should a Streaming Video Model Remember?

arXiv:2606. 16353v1 Announce Type: cross Abstract: Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets.

By Haonan Ge, Yiwei Wang, Hang Wu, Yujun Cai

multimodal benchmarks

arXiv AI

Jun 16

MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild

arXiv:2606. 16731v1 Announce Type: cross Abstract: Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios.

By Haotian Qi, Gabriel Skantze

robotics multimodal

arXiv AI

Jun 16

DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing

arXiv:2606. 15796v1 Announce Type: cross Abstract: Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits.

By Artyom Mazur, Nina Konovalova, Aibek Alanov

llms diffusion efficiency multimodal benchmarks safety

arXiv AI

Jun 16

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

arXiv:2606. 16092v1 Announce Type: cross Abstract: Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements.

By Young Rok Jang, Hyesoo Kong, Kyunghwan An, Jae Sub Huh, Gyeonghun Kim, Stanley Jungkyu Choi

llms fine-tuning multimodal

arXiv AI

Jun 16

Semantics-Enhanced Retrieval-Augmented Time Series Forecasting

arXiv:2606. 14941v1 Announce Type: new Abstract: Time series forecasting models often benefit from historical patterns.

By Shiqiao Zhou, Zipeng Wu, Holger Sch\"oner, Edouard Fouch\'e, IAG Wilson, Shuo Wang

rag multimodal benchmarks

arXiv AI

Jun 16

Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech

arXiv:2606. 14750v1 Announce Type: cross Abstract: Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding.

By Adarsh Arigala, Arjun Gangwar, S Umesh, Yova Kementchedjhieva

rag fine-tuning multimodal

arXiv AI

Jun 16

Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments

arXiv:2505. 19699v2 Announce Type: replace-cross Abstract: Federated Learning (FL) is a decentralized machine learning paradigm that enables clients to collaboratively train models while preserving data privacy.

By Junming Liu, Yanting Gao, Yuqi Li, Siyuan Meng, Yifei Sun, Aoqi Wu, Yirong Chen, Ding Wang, Shiping Wen

efficiency multimodal benchmarks safety

arXiv AI

Jun 16

Active Inference with a Self-Prior in the Mirror-Mark Task

arXiv:2604. 09673v2 Announce Type: replace-cross Abstract: The mirror self-recognition test evaluates whether a subject touches a mark on its own body that is visible only in a mirror, and is widely used as an indicator of self-awareness.

By Dongmin Kim, Hoshinori Kanazawa, Yasuo Kuniyoshi

llms multimodal

arXiv AI

Jun 16

FlowMPC: Improving Flow Matching policies with World Models

arXiv:2606. 16286v1 Announce Type: cross Abstract: Flow Matching (FM) is a powerful approach for behavior cloning in multimodal action spaces [Jiang et al.

By Chandon Hamel

diffusion robotics multimodal

arXiv AI

Jun 16

Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models

arXiv:2606. 16808v1 Announce Type: new Abstract: While Large Reasoning Models (LRMs) excel at complex tasks, they remain highly vulnerable to sophisticated jailbreaks and direct harmful queries.

By Ke Miao, Jiaxin Li, Hongliang Chen, Yuke Hu, Zhan Qin

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 16

A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT

arXiv:2606. 16991v1 Announce Type: cross Abstract: Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization, yet it carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload.

By Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy, Aya Yassin, Concetto Spampinato, Karim Lekadir, Xiaomeng Li, Marawan Elbatel

multimodal benchmarks

arXiv AI

Jun 16

Virtual Sensing to Enable Real-Time Monitoring of Inaccessible Locations & Unmeasurable Parameters

arXiv:2412. 00107v2 Announce Type: replace-cross Abstract: Real-time monitoring of safety-critical interior states remains an open problem in energy systems where physical instrumentation is infeasible.

By Kazuma Kobayashi, Farid Ahmed, Jaewan Park, Subhankar Sarkar, Souvik Chakraborty, Syed Bahauddin Alam

multimodal safety

arXiv AI

Jun 16

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

arXiv:2606. 15749v1 Announce Type: cross Abstract: Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics.

By Maonan Wang, Zhengyan Huang, Kemou Jiang, Yuhang Fu, Jiayue Zhu, Yuxin Cai, Xingchen Zou, Qiaosheng Zhang, Yi Yu, Ding Wang, Xi Chen, Ben M. Chen, Yuxuan Liang, Zhiyong Cui, Man On Pun, Yirong Chen

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 16

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

arXiv:2606. 15160v1 Announce Type: cross Abstract: Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years.

By David Huang, Lianlei Shan

llms diffusion reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jun 16

X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining

arXiv:2606. 14752v1 Announce Type: cross Abstract: Modern Vision-Language-Action (VLA) models must bridge pretrained vision-language reasoning and precise continuous robot control.

By Xirui Kang, Yanpei Shi, Lucy Liang, Roy Gan, Dongxiu Liu, Pushi Zhang, Danpeng Chen, Xiaoyi Qin, Yinan Zheng, Jinliang Zheng, Hao Wang, Xianyuan Zhan, Hang Su

nlp robotics efficiency multimodal safety

arXiv AI

Jun 16

Dual-Uncertainty Guided Policy Learning for Multimodal Reasoning

arXiv:2510. 01444v3 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards (RLVR) has advanced reasoning capabilities in multimodal large language models.

By Rui Liu, Dian Yu, Tong Zheng, Runpeng Dai, Zongxia Li, Wenhao Yu, Zhenwen Liang, Linfeng Song, Haitao Mi, Pratap Tokekar, Dong Yu

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jun 16

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

arXiv:2605. 27284v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed.

By Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, Yixuan Chen, Yingming Zheng, Shuai Bai, Tao Yu

robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 16

HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation

arXiv:2306. 11252v2 Announce Type: replace-cross Abstract: We introduce HK-LegiCoST, a new three-way parallel corpus of Cantonese-English translations, containing 600+ hours of Cantonese audio, its standard traditional Chinese transcript, and English translation, segmented and aligned at the sentence level.

By Cihan Xiao, Henry Li Xinyuan, Jinyi Yang, Dongji Gao, Matthew Wiesner, Kevin Duh, Sanjeev Khudanpur

computer-vision multimodal safety

arXiv AI

Jun 16

Driving, Fast or Slow? Neuro-Symbolic Guidance for Motion Prediction in Multi-Modal Ground Mobility

arXiv:2606. 15251v1 Announce Type: cross Abstract: Accurate and interpretable motion prediction for heterogeneous traffic spaces, including pedestrians, bicycles, cars, and trucks, is essential for safe autonomous navigation.

By Simon Kohaut, Felix Divo, Julius Hahnewald, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami

agents multimodal benchmarks

Get the daily briefing