Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,277 stories · RSS feed

arXiv Machine Learning

Jun 16

HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translation

arXiv:2306. 11252v2 Announce Type: replace-cross Abstract: We introduce HK-LegiCoST, a new three-way parallel corpus of Cantonese-English translations, containing 600+ hours of Cantonese audio, its standard traditional Chinese transcript, and English translation, segmented and aligned at the sentence level.

By Cihan Xiao, Henry Li Xinyuan, Jinyi Yang, Dongji Gao, Matthew Wiesner, Kevin Duh, Sanjeev Khudanpur

computer-vision multimodal safety

arXiv AI

Jun 16

Lect\=uraAgents: A Multi-Agent Framework for Adaptive Personalized AI-Assisted Learning and Embodied Teaching

arXiv:2606. 16428v1 Announce Type: cross Abstract: Effective personalized AI-assisted learning demands systems that can not only generate accurate learner-specific educational materials, but also dynamically adapt their instruction to diverse learners.

By Jaward Sesay, Yue Yu, Siwei Dong, Yemin Shi, Guangyao Chen, B\"orje F. Karlsson

agents computer-vision robotics multimodal safety

arXiv AI

Jun 16

UniT: Unified Multimodal Chain-of-Thought Test-time Scaling

arXiv:2602. 12279v2 Announce Type: replace-cross Abstract: Unified models can handle both multimodal understanding and generation within a single architecture, yet they typically operate in a single pass without iteratively refining their outputs.

By Leon Liangyu Chen, Haoyu Ma, Zhipeng Fan, Ziqi Huang, Animesh Sinha, Xiaoliang Dai, Jialiang Wang, Zecheng He, Jianwei Yang, Chunyuan Li, Junzhe Sun, Chu Wang, Serena Yeung-Levy, Felix Juefei-Xu

llms agents multimodal

arXiv AI

Jun 16

XAI-Grounded Explanation Generation for Speech Deepfake Detection with Training-Free Multimodal Large Language Models

arXiv:2606. 16137v1 Announce Type: cross Abstract: Speech deepfake detection (SDD) systems require trustworthy explanations for reliable decision-making.

By Yupei Li, Qiyang Sun, Xiaoliang Wu, Chenxi Wang, Berrak Sisman, Bj\"orn W. Schuller

llms multimodal

arXiv AI

Jun 16

MapDream: Task-Driven Map Learning for Vision-Language Navigation

arXiv:2602. 00222v3 Announce Type: replace-cross Abstract: Vision-Language Navigation (VLN) requires agents to follow natural language instructions in partially observed 3D environments, motivating map representations that aggregate spatial context beyond local perception.

By Guoxin Lian, Shuo Wang, Yucheng Wang, Yongcai Wang, Maiyue Chen, Kaihui Wang, Bo Zhang, Zhizhong Su, Deying Li, Zhaoxin Fan

agents fine-tuning multimodal benchmarks

arXiv AI

Jun 16

TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting

arXiv:2606. 16173v1 Announce Type: new Abstract: High-quality time series forecasting is pivotal for real-world decision-making.

By Zhi Chen, Yuxuan Wang, Jialong Wu, Yong Liu, Haoran Zhang, Xingjian Su, Jianmin Wang, Mingsheng Long

llms multimodal benchmarks

arXiv AI

Jun 16

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

arXiv:2605. 27284v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed.

By Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, Yixuan Chen, Yingming Zheng, Shuai Bai, Tao Yu

robotics multimodal benchmarks safety

arXiv AI

Jun 16

Action with Visual Primitives

arXiv:2605. 22183v3 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for generalist robotic manipulation.

By Weilong Guo, Yuchen Wang, Renping Zhou, Yunfeng Zhang, Rui Fang, Yuyang Pang, Wenda Xu, Gao Huang

robotics multimodal

arXiv AI

Jun 16

Thinking with Visual Grounding

arXiv:2606. 16122v1 Announce Type: new Abstract: Visual thinking should not only sound right; it should show its evidence.

By Junkai Zhang, Yihe Deng, Kai-Wei Chang, Wei Wang

llms agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 16

Scaling Human and G2P Supervision for Robust Phonetic Transcription

arXiv:2606. 16019v1 Announce Type: cross Abstract: Expert phonetic annotation is costly, especially for non-standard dialects and atypical speech.

By Alexander Metzger, Aruna Srivastava, Ruslan Mukhamedvaleev

multimodal benchmarks

arXiv AI

Jun 16

Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

arXiv:2606. 15038v1 Announce Type: new Abstract: Accurate time-to-event (TTE) prediction from multimodal clinical data remains challenging due to modality imbalance and distribution shift.

By Zhemin Zhang, Weijie Chen, David Le, Amara Tariq, Alex Wallace, Matthew Stib, Juan Maria Farina, Chadi Ayoub, Reza Arsanjani, Imon Banerjee

diffusion multimodal safety

arXiv AI

Jun 16

Active Inference with a Self-Prior in the Mirror-Mark Task

arXiv:2604. 09673v2 Announce Type: replace-cross Abstract: The mirror self-recognition test evaluates whether a subject touches a mark on its own body that is visible only in a mirror, and is widely used as an indicator of self-awareness.

By Dongmin Kim, Hoshinori Kanazawa, Yasuo Kuniyoshi

llms multimodal

arXiv AI

Jun 16

Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments

arXiv:2505. 19699v2 Announce Type: replace-cross Abstract: Federated Learning (FL) is a decentralized machine learning paradigm that enables clients to collaboratively train models while preserving data privacy.

By Junming Liu, Yanting Gao, Yuqi Li, Siyuan Meng, Yifei Sun, Aoqi Wu, Yirong Chen, Ding Wang, Shiping Wen

efficiency multimodal benchmarks safety

arXiv AI

Jun 16

EyeMVP: OCT-Informed Fundus Representation Learning via Paired CFP--OCT Pretraining

arXiv:2606. 15129v1 Announce Type: cross Abstract: Color fundus photography (CFP) is the mainstay for large-scale retinal screening, yet its diagnostic capacity is constrained by the lack of depth-resolved structural information.

By Zhuo Deng, Ruiheng Zhang, Ziheng Zhang, Weihao Gao, Yitong Li, Qian Wang, Lei Shao, Jiaoyue Dong, Zhixi Zeng, Lijian Fang, Haibo Wang, Xiaobin Lin, Tao Liu, Zhicheng Du, Zhengwei Zhang, Lin Yang, Zheng Gong, Xinyu Zhao, Zhenquan Wu, Fang Li, Zhiguang Zhou, Guoming Zhang, Sun Jing, Han Lv, Wenbin We, Lan Ma

computer-vision multimodal

arXiv Machine Learning

Jun 16

DiRecT: Safe Diffusion-Based Planning via Receding-Horizon Denoising

arXiv:2606. 15359v1 Announce Type: new Abstract: Diffusion models have emerged as powerful tools for planning and control by learning multimodal distributions over actions and trajectories.

By Paolo Giaretta, Zeyang Li, Navid Azizan

diffusion multimodal benchmarks safety

arXiv Machine Learning

Jun 16

PHINN: Persistent Homology Inspired Neural Network for Rare-Event Time Series Generation

arXiv:2606. 15452v1 Announce Type: new Abstract: Rare events in time series are critical to model but hard to learn due to data scarcity.

By Emre Yusuf, Ren Takahashi, Jayabrata Bhaduri

rag diffusion multimodal benchmarks safety

arXiv AI

Jun 16

Automated jailbreak attack targeting multiple defense strategies

arXiv:2606. 16751v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks.

By Qi Wang, Chengcheng Wan, Weijia He, Yanqing Li, Hanqi Sun, Xiaodong Gu, Jiangtao Wang

llms multimodal safety

arXiv AI

Jun 16

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

arXiv:2606. 16092v1 Announce Type: cross Abstract: Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements.

By Young Rok Jang, Hyesoo Kong, Kyunghwan An, Jae Sub Huh, Gyeonghun Kim, Stanley Jungkyu Choi

llms fine-tuning multimodal

arXiv AI

Jun 16

PAL-Bench: Evidence-Grounded Profile Reconstruction from Longitudinal Personal Albums

arXiv:2606. 16175v1 Announce Type: new Abstract: Longitudinal personal albums are weak-schema multimodal databases: noisy perceptual records whose key facts require joins across faces, text, timestamps, locations, and repeated events.

By Qiwei Yan, Zhiqiang Yuan, Zexi Jia, Nanxing Hu, Kailin Lyu, Jie Zhou, Jinchao Zhang

agents nlp multimodal benchmarks safety

arXiv AI

Jun 16

Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

arXiv:2606. 16494v1 Announce Type: cross Abstract: Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-scale knowledge base.

By Jieyuan Liu, Jianyang Gu, Shijie Chen, Jefferson Chen, Zhen Wang

llms rag nlp multimodal benchmarks safety

Get the daily briefing