Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,869 stories · RSS feed

arXiv AI

Jul 24

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

arXiv:2602. 19313v2 Announce Type: replace-cross Abstract: General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior.

By Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna

llms robotics multimodal benchmarks

arXiv Machine Learning

Jul 24

Wireless TokenCom: RL-Based Tokenizer Agreement for Multi-User Wireless Token Communications

arXiv:2602. 12338v2 Announce Type: replace Abstract: Token Communications (TokenCom) has recently emerged as an effective new paradigm, where tokens are the unified units of multimodal communications and computations, enabling efficient digital semantic- and goal-oriented communications in future wireless networks.

By Farshad Zeinali, Mahdi Boloursaz Mashhadi, Rahim Tafazolli

diffusion nlp reinforcement-learning multimodal

arXiv Machine Learning

Jul 24

X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

arXiv:2607. 21550v1 Announce Type: new Abstract: While large audio-language models have achieved remarkable progress in auditory perception, they still lag behind text-based large language models in deep logical reasoning, primarily due to the scarcity of high-quality audio reasoning data.

By Dongjie Fu, Di Cao, Xize Cheng, Zihan Zhang, Wenxu Jia, Yifu Chen, Shengpeng Ji, Yu Zhang, Tao Jin

llms efficiency multimodal safety

arXiv AI

Jul 24

GS-Agent: Creating 4D Physical Worlds With Generative Simulation

arXiv:2607. 21522v1 Announce Type: cross Abstract: Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging.

By Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan

agents robotics fine-tuning multimodal

arXiv Machine Learning

Jul 24

Semantics at an Angle: When Cosine Similarity Works Until It Doesn't

arXiv:2504. 16318v3 Announce Type: replace Abstract: Cosine similarity is a standard comparison rule for learned representations in information retrieval, natural language processing, computer vision, and multimodal learning.

By Kisung You

rag computer-vision nlp multimodal safety

arXiv AI

Jul 24

Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUs

arXiv:2607. 21042v1 Announce Type: new Abstract: Autoregressive text-to-speech models achieve strong naturalness but suffer from slow inference due to sequential token generation, limiting their deployment in production applications that require low latency.

By Muyang Du, Shuang Yu, Junjie Lai

llms diffusion efficiency multimodal benchmarks

arXiv AI

Jul 24

OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

arXiv:2602. 12304v5 Announce Type: replace-cross Abstract: Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts.

By Maomao Li, Zhen Li, Kaipeng Zhang, Guosheng Yin, Zhifeng Li, Dong Xu

diffusion fine-tuning multimodal

arXiv AI

Jul 24

OpenForgeRL: Train Harness-native Agents in Any Environment

arXiv:2607. 21557v1 Announce Type: new Abstract: Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems.

By Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 24

Multimodality Stacking with Blockwise missing values and application to the PIONeeR biomarkers study for prediction of resistance to immunotherapy

arXiv:2605. 25050v2 Announce Type: replace-cross Abstract: Integrating multimodal datasets in clinical oncology is frequently hindered by high dimensionality and blockwise missingness, where entire data sources are unavailable for specific patient subsets.

By Mohamed Boussena, Florence Monville, Jacques Fieschi-Meric, Frederic Vely, Pierre Milpied, Julien Mazieres, Maurice Perol, Eric Vivier, Laurent Greillier, Fabrice Barlesi, Sebastien Benzekry

multimodal

arXiv AI

Jul 24

Multimodal Pretraining for Generalizable EEG Representation Learning

arXiv:2607. 21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks.

By Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi

llms rag computer-vision fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jul 24

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.

By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

llms rag multimodal benchmarks safety

arXiv AI

Jul 24

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

arXiv:2607. 21013v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks and lifted MER to a new level that is complex emotion understanding with advanced video understanding abilities and natural language description.

By Lihuang Fang, Yuchen Zou, kebin Jin, Jinghui Qin

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 24

Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts

arXiv:2607. 20462v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output with watermarking.

By Melanie Rieff, Robin Staab, Thibaud Gloaguen, Stefan Hegselmann, Martin Vechev

llms multimodal benchmarks safety

arXiv AI

Jul 24

HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving

arXiv:2607. 20988v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving.

By Quanfu Yu, Xian Wu, Hao Xu, Liulong Ma

agents robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation

arXiv:2607. 21371v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) leverages textual semantics to segment objects beyond predefined categories.

By Sung-Hoon Yoon, Hoyong Kwon, Changgyoon Oh, Kuk-Jin Yoon

rag computer-vision multimodal benchmarks safety

arXiv AI

Jul 24

SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning

arXiv:2607. 20511v1 Announce Type: new Abstract: Multimodal Continual Instruction Tuning (MCIT) is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving a sequence of downstream tasks.

By Keonhee Park, Gunhee Kim

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 24

Vision-Language-Policy Model for Dynamic Robot Task Planning

arXiv:2512. 19178v2 Announce Type: replace-cross Abstract: Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics.

By Jin Wang, Kim Tien Ly, Jacques Cloete, Jin Jin, Nikos Tsagarakis, Ioannis Havoutis

llms agents robotics fine-tuning multimodal

arXiv AI

Jul 24

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

arXiv:2607. 21300v1 Announce Type: cross Abstract: Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations.

By Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, Davide Talon, Elisa Ricci

llms nlp fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance

arXiv:2511. 12997v2 Announce Type: replace Abstract: Multimodal LLM-powered agents have recently demonstrated impressive capabilities in web navigation, enabling agents to complete complex browsing tasks across diverse domains.

By Genglin Liu, Shijie Geng, Sha Li, Hejie Cui, Sarah Zhang, Xin Liu, Tianyi Liu

llms agents multimodal benchmarks

arXiv AI

Jul 24

When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion

arXiv:2607. 20543v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.

By Todd Zhou

agents reinforcement-learning multimodal benchmarks

Get the daily briefing