Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,396 stories · RSS feed

arXiv AI

Jul 1

Human-Agent Collaborative Paper-to-Page Crafting

arXiv:2510. 19600v2 Announce Type: replace-cross Abstract: In the quest for scientific progress, communicating research is as vital as the discovery itself.

By Qianli Ma, Siyu Wang, Yilin Chen, Yinhao Tang, Yixiang Yang, Chang Guo, Bingjie Gao, Zhening Xing, Yanan Sun, Zhipeng Zhang

agents multimodal benchmarks safety

arXiv AI

Jul 1

Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning

arXiv:2606. 31825v1 Announce Type: cross Abstract: Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences.

By Junha Jung, Minbyul Jeong, Suhyeon Lim, Sungwook Jung, Jaehoon Yun, Taeyun Roh, Mujeen Sung, Jaewoo Kang

llms nlp reinforcement-learning multimodal benchmarks

arXiv AI

Jul 1

RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization

arXiv:2606. 31694v1 Announce Type: cross Abstract: For robots manipulating open-world objects, tactile representations must generalize to unseen materials.

By Jingbo He, Michael F\"arber, Roberto Calandra

rag robotics multimodal

arXiv AI

Jul 1

Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

arXiv:2606. 31599v1 Announce Type: cross Abstract: Vision-language models (VLMs) combining reinforcement learning (RL) ignite remarkable progress in multimodal reasoning, yet still struggle with medical images, which typically exhibit extremely sparse visual evidence to inform clinical decision-making.

By Kaitao Chen, Weiqian Zhao, Jiamin Wu, Qihao Zheng, Shangquan Sun, Chunfeng Song, Xiaosong Wang, Mu Zhou, Mianxin Liu

llms nlp reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jul 1

UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation

arXiv:2606. 31451v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) have shown great promise in integrating understanding and generation across diverse modalities.

By Jiahang Tu, Fengyu Yang, Chenyang Ma, Xihang Yu, Ziyao Zeng, Shaokai Wu, Hanbin Zhao, Zhi Tao, Chao Zhang, Hui Qian, Alex Wong

multimodal benchmarks safety

arXiv AI

Jul 1

PGUDA: Pressure-Guided Unsupervised Domain Adaptation with Cross-Modal Knowledge Distillation for sEMG-Based Gesture Recognition

arXiv:2606. 31349v1 Announce Type: cross Abstract: Surface electromyography (sEMG)-based gesture recognition has emerged as a promising technology for natural human-computer interaction.

By Yurui Liu, Xiao-Cong Zhong, Qisong Wang, Xuefu Wang, Dan Liu, Jinwei Sun

fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jul 1

FedLAB: Traceable Semantic Codebooks for Federated Multimodal Graph Foundation Learning

arXiv:2606. 32016v1 Announce Type: new Abstract: Multimodal graph foundation models aim to learn reusable knowledge from graphs enriched with text, images, attributes, and relational topology, thereby supporting diverse graph-centric and modality-centric tasks.

By Zekai Chen, Kairui Yang, Xuaner Chen, Xunkai Li, Xun Wu, Rong-Hua Li, Guoren Wang

rag multimodal benchmarks safety

arXiv AI

Jul 1

Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents

arXiv:2606. 31270v1 Announce Type: cross Abstract: Computer-use agents, which leverage multimodal large language models (MLLMs) to operate computers and complete tasks, have attracted significant attention for their utility and versatility.

By Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt, Serena Yeung-Levy, Yuhui Zhang

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jul 1

TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models

arXiv:2606. 31976v1 Announce Type: new Abstract: Human-labeled data are widely used as reference annotations in ML, despite known variability across annotators in many expert-driven domains.

By Shiyi Chen, Nicholas Saban, Collin Hargreaves, Huiqi Wang

llms agents multimodal safety

arXiv AI

Jul 1

The Label Imitation Game: Turing Test Network for Zero-Shot Pseudo-Label Pruning

arXiv:2606. 30875v1 Announce Type: cross Abstract: Foundation model pseudo-labeling - labeling data strictly via zero-shot inference - enables massive scale, but performance is undermined by hallucinations that evade standard thresholds.

By Brent A. Griffin, Jason J. Corso

llms computer-vision fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jul 1

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

arXiv:2606. 31329v1 Announce Type: cross Abstract: Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation.

By Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

llms robotics multimodal benchmarks

arXiv AI

Jul 1

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

arXiv:2606. 30686v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks.

By Taozhao Chen, Ian Manchester, Huaming Chen

llms robotics multimodal benchmarks

arXiv AI

Jul 1

ELEVATE: Designing Human-Centered GenAI Virtual Tutors for Scalable and Inclusive Education

arXiv:2606. 30662v1 Announce Type: cross Abstract: The advent of Generative Artificial Intelligence (GenAI), and in particular Large Language Models (LLMs), is reshaping educational practice, while intensifying ethical debate about its adoption.

By Lorenzo Stacchio, Michele Giordano, Daniele Berardini, Primo Zingaretti, Emanuele Frontoni

llms robotics multimodal safety

arXiv AI

Jul 1

ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection

arXiv:2606. 30646v1 Announce Type: cross Abstract: Speech recruits the same executive, attentional, and working memory processes underlying instrumental activities of daily living, or IADLs, providing a non-invasive proxy for cognitive assessment.

By Chukwuemeka Ugwu, Oluwafemi Richard Oyeleke

llms rag multimodal benchmarks

arXiv AI

Jul 1

Cross-Modal Hierarchical Fusion for from Multi-Sensor Ground Observation

arXiv:2606. 30647v1 Announce Type: cross Abstract: Dense volumetric reconstruction of cloud microphysical fields from sparse ground-based instruments remains an open problem, largely because the available measurements are heterogeneous in both modality and spatial coverage.

By Xinze Zhang

multimodal benchmarks

arXiv AI

Jul 1

Evo-PI: Aligning Medical Reasoning via Evolving Principle-Guided Supervision

arXiv:2606. 31800v1 Announce Type: new Abstract: Despite recent progress, the reasoning capabilities of large multimodal language models (MLLMs) remain fundamentally constrained by static supervision, where fixed prompts, rules, or reward models provide non-adaptive guidance throughout training.

By Xianda Zheng, Huan Gao, Meng-Fen Chiang, Michael Witbrock, Kaiqi Zhao, Shangyang Li

llms nlp multimodal benchmarks safety

arXiv AI

Jul 1

Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

arXiv:2606. 31711v1 Announce Type: new Abstract: Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) models.

By Yuanhao Ban, Tong Xie, Sohyun An, Yunqi Hong, Evan Frick, I-Hung Hsu, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

diffusion multimodal benchmarks

arXiv AI

Jul 1

Surprise as a Signal for Plasticity and Metacognition

arXiv:2606. 31495v1 Announce Type: new Abstract: We study a single idea across two settings: that a prediction-error signal, computed by a small predictor over the latent space of a frozen encoder, can serve both as a gate on plasticity and as a substrate for metacognition.

By Louis Mouchon

llms diffusion multimodal benchmarks

arXiv AI

Jul 1

Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping

arXiv:2606. 31200v1 Announce Type: new Abstract: Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM-based methods rely on visual similarity for object matching, neglecting physical affordances such as handle graspability and material fragility, and operate open-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse.

By Tao Chen, Lizheng Liu, Jiaxu Wang, Ziyue Jiang, Ruiqi Tian, JiGuang Huo, Zhongxue Gan

llms rag agents robotics multimodal benchmarks

arXiv Machine Learning

Jul 1

Structured SIR: Efficient and Expressive Importance-Weighted Inference for High-Dimensional Image Registration

arXiv:2603. 17415v2 Announce Type: replace-cross Abstract: Image registration is an ill-posed dense vision task, where multiple solutions achieve similar loss values, motivating probabilistic inference.

By Ivor J. A. Simpson, Neill D. F. Campbell

multimodal

Get the daily briefing