Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,954 stories · RSS feed

arXiv AI

Jul 24

Monkey King Bang: A Unified Scientific Multimodal Foundation Model

arXiv:2607. 20557v1 Announce Type: cross Abstract: Scientific discovery is increasingly shifting from isolated disciplines to multi-domain reasoning, and AI for science faces a similar transition.

By Hesen Chen, Xinyu Su, Xiaomeng Yang, Yuetan Lin, Zixiong Yang, Junyi An, Fenglei Cao, Yifeng Jiao, Yunqi Zhang, Yuan Cheng, Zhiyu Tan, Hao Li, Libo Wu, Yuan Qi

llms computer-vision multimodal benchmarks

arXiv AI

Jul 24

Constrained latent state modeling: A unifying perspective on representation learning under competing constraints

arXiv:2605. 15995v2 Announce Type: replace-cross Abstract: Learning latent representations from complex data is central to modern machine learning, spanning temporal, multimodal, and partially observed systems.

By Gwenol\'e Quellec

multimodal benchmarks

arXiv AI

Jul 24

Multi-Task Learning for Heterogeneous Prediction from Video Game State with Transfer Learning

arXiv:2607. 21290v1 Announce Type: cross Abstract: Multi-task learning (MTL) is a promising approach for prediction tasks derived from video game state data, as modern game telemetry provides multiple related supervision signals from the same structured observations.

By Jonas Pech\'e, Aliaksei Tsishurou, Alexander Zap, G\"unter Wallner

fine-tuning multimodal

arXiv AI

Jul 24

OpenForgeRL: Train Harness-native Agents in Any Environment

arXiv:2607. 21557v1 Announce Type: new Abstract: Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems.

By Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 24

Multimodal Pretraining for Generalizable EEG Representation Learning

arXiv:2607. 21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks.

By Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi

llms rag computer-vision fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

Enhancing Explainable Cardiac Diagnosis with Guide-Grounded Multimodal LLMs

arXiv:2607. 20814v1 Announce Type: new Abstract: The electrocardiogram (ECG) is a cornerstone of cardiac as- sessment, yet clinical deployment of deep learning models remains con- strained by limited interpretability and the hallucination risk of large language models (LLMs).

By Hai-Nam Duy Vuong, Duy-Anh Bui, Trong-Nghia Nguyen, Kim-Ngan Thi Nguyen, Trang Mai Xuan, Tien-Cuong Nguyen, Van-Dem Pham, Thien Van Luong

llms multimodal safety

arXiv AI

Jul 24

DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation

arXiv:2607. 21371v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) leverages textual semantics to segment objects beyond predefined categories.

By Sung-Hoon Yoon, Hoyong Kwon, Changgyoon Oh, Kuk-Jin Yoon

rag computer-vision multimodal benchmarks safety

arXiv AI

Jul 24

M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data

arXiv:2607. 21343v1 Announce Type: cross Abstract: Integrating heterogeneous biomedical data, including clinical metadata, histopathology images, and molecular profiles, is crucial for comprehensive disease understanding.

By Francesca Pia Panaccione, Carlo Sgaravatti, Marco Venere

rag multimodal safety

arXiv Machine Learning

Jul 24

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.

By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jul 24

X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

arXiv:2607. 21550v1 Announce Type: new Abstract: While large audio-language models have achieved remarkable progress in auditory perception, they still lag behind text-based large language models in deep logical reasoning, primarily due to the scarcity of high-quality audio reasoning data.

By Dongjie Fu, Di Cao, Xize Cheng, Zihan Zhang, Wenxu Jia, Yifu Chen, Shengpeng Ji, Yu Zhang, Tao Jin

llms efficiency multimodal safety

arXiv Machine Learning

Jul 24

Semantics at an Angle: When Cosine Similarity Works Until It Doesn't

arXiv:2504. 16318v3 Announce Type: replace Abstract: Cosine similarity is a standard comparison rule for learned representations in information retrieval, natural language processing, computer vision, and multimodal learning.

By Kisung You

rag computer-vision nlp multimodal safety

arXiv AI

Jul 24

Robostral Navigate

arXiv:2607. 20785v1 Announce Type: cross Abstract: Deploying navigation systems at scale requires a recipe that minimizes sensor assumptions, generalizes across robot embodiments, and trains efficiently.

By Arjun Majumdar, Avinash Sooriyarachchi, Benjamin Tibi, Chris Bamford, Elliot Chane-Sane, Guillaume Lample, Khyathi Raghavi Chandu, Ludovic Ho Fuh, Mathieu Poiree, Olivier Duchenne, Rosalie Millner, Srijan Mishra, Theo Cachet, Thomas Chabal

llms reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jul 24

GS-Agent: Creating 4D Physical Worlds With Generative Simulation

arXiv:2607. 21522v1 Announce Type: cross Abstract: Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging.

By Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan

agents robotics fine-tuning multimodal

arXiv AI

Jul 24

MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

arXiv:2508. 05502v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) perform strongly in high-resource languages, yet often produce fluent but culturally "thin" descriptions in low-resource settings.

By Yufei Gao, Jiaying Fei, Nuo Chen, Ruirui Chen, Guohang Yan, Yunshi Lan, Botian Shi

llms fine-tuning multimodal safety

arXiv AI

Jul 24

Beyond SBDD: Geometric Deep Learning in Polypharmacology and Multi-target Drug Design

arXiv:2607. 20550v1 Announce Type: cross Abstract: The traditional "one drug, one target" paradigm of structure-based drug design (SBDD) frequently proves inadequate for treating multifactorial diseases such as cancer and neurodegenerative disorders, owing to compensatory signaling pathways and the emergence of drug resistance.

By Tianming Han, Zhijie Pan, Wenchi Ge, Qi Zhao

rag diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jul 24

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

arXiv:2602. 19313v2 Announce Type: replace-cross Abstract: General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior.

By Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna

llms robotics multimodal benchmarks

arXiv Machine Learning

Jul 24

Best-of-Evidence: Best-of-N Selection under Partial Verification

arXiv:2607. 20950v1 Announce Type: new Abstract: BoN improves model outputs by sampling several candidates and selecting one with a proxy score, but it assumes that complete candidates can be evaluated reliably.

By Cenwei Zhang, Teng Fang, Yuxia Wang, Derek Li, Bryan Dai, Lei You

multimodal

arXiv AI

Jul 24

SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning

arXiv:2607. 20511v1 Announce Type: new Abstract: Multimodal Continual Instruction Tuning (MCIT) is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving a sequence of downstream tasks.

By Keonhee Park, Gunhee Kim

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 24

VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory

arXiv:2603. 04910v2 Announce Type: replace-cross Abstract: Imitation learning from human demonstrations has achieved significant success in robotic control, yet most visuomotor policies still condition on single-step observations or short-context histories, making them struggle with non-Markovian tasks that require long-term memory.

By Yuheng Lei, Zhixuan Liang, Hongyuan Zhang, Ping Luo

llms rag diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jul 24

Environment-Aware Channel Inference via Cross-Modal Flow: From Multimodal Sensing to Wireless Channels

arXiv:2512. 04966v2 Announce Type: replace-cross Abstract: Accurate channel state information (CSI) underpins reliable and efficient wireless communication.

By Guangming Liang, Mingjie Yang, Dongzhu Liu, Paul Henderson, Lajos Hanzo

diffusion multimodal benchmarks safety

Get the daily briefing