Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,393 stories · RSS feed

arXiv AI

Jun 10

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

arXiv:2606. 10819v1 Announce Type: cross Abstract: RS-MLLMs enable natural-language understanding and spatial reasoning over earth observation imagery.

By Miaoxin Cai, Guanqun Wang, Wei Zhang, Guangyao Zhou, Yin Zhuang, Tong Zhang, Hao Wang, He Chen, Jun Li

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 10

MemVenom: Triggered Poisoning of Multimodal Memories in Web Agents

arXiv:2606. 10742v1 Announce Type: cross Abstract: External memory has become a core component of modern web agents, enabling long-horizon reasoning through the retrieval of past experiences.

By Yv Zhang, Hao Sun, Hao Fang, Kuofeng Gao, Fan Mo, Bin Chen, Shu-Tao Xia, Yaowei Wang

llms agents computer-vision multimodal safety

arXiv Machine Learning

Jun 10

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

arXiv:2606. 11190v1 Announce Type: new Abstract: Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality.

By Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets, Randall Balestriero

multimodal benchmarks safety

arXiv AI

Jun 10

Architect-Ant: Editable Automatic Furnishing of Architectural Floor Plans

arXiv:2606. 10953v1 Announce Type: new Abstract: Furnished floor plans are fundamental to real estate visualization, interior design, and architectural workflows.

By Fedor Rodionov, Aleksandar Cvejic, Michael Birsak, John Femiani, Peter Wonka

llms fine-tuning multimodal safety

arXiv AI

Jun 10

Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation

arXiv:2606. 10368v1 Announce Type: cross Abstract: Speech-to-text (S2T) systems for recognition (ASR) and translation (S2TT) typically generate discrete text tokens.

By Xuanchen Li, Tianrui Wang, Yuheng Lu, Zikang Huang, Yu Jiang, Chenghan Lin, Chenrui Cui, Ziyang Ma, Xingyu Ma, Chunyu Qiang, Guochen Yu, Xie Chen, Longbiao Wang, Jianwu Dang

diffusion multimodal

arXiv AI

Jun 10

Co-GLANCE: Uncertainty-Aware Active Perception for Heterogeneous Robot Teaming

arXiv:2606. 09919v1 Announce Type: cross Abstract: Perceptual uncertainty is a central challenge for heterogeneous robot teams operating in unstructured outdoor environments, where no single viewpoint affords reliable scene understanding.

By Michal P. Podolinsky, Neel P. Bhatt, Pranay Samineni, Rohan Siva, Christian Ellis, Ufuk Topcu

llms computer-vision robotics efficiency multimodal

arXiv AI

Jun 10

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

arXiv:2606. 10572v1 Announce Type: new Abstract: External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence.

By Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee

llms rag nlp efficiency multimodal benchmarks

arXiv AI

Jun 10

Exploration of Foundation Model-Based Robots in Patient and Elderly Care

arXiv:2606. 10208v1 Announce Type: cross Abstract: Demand for older-adult and patient care is growing rapidly as populations age worldwide.

By Zhiwen Qiu, Wei Liu, Yuexing Hao

agents robotics multimodal safety

arXiv AI

Jun 10

MetaPlate: Counterfactual-Guided RAG-LLM Tool for Personalized Food Recommendation and Hyperglycemia Prevention

arXiv:2606. 10120v1 Announce Type: cross Abstract: Postprandial hyperglycemia is a key risk factor for metabolic disorders; however, existing dietary guidance is often static, impractical, and insufficiently personalized, providing recommendations that are difficult to follow or not impactful.

By Asiful Arefeen, Carol Johnston, Hassan Ghasemzadeh

llms rag multimodal safety

arXiv AI

Jun 10

Soul Computing: A Theoretical Framework and Technical Architecture for Intelligent Agents with Independent Consciousness

arXiv:2606. 10413v1 Announce Type: new Abstract: Breakthroughs in large language models and multimodal generation technologies have propelled the digital reconstruction of human mental traits, emotional patterns, and long-term memory from science fiction toward engineering practice.

By Jinshan Zhang, Xishi Zhou, Qiu Peng, Jianwei Yin

llms agents multimodal

arXiv AI

Jun 10

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

arXiv:2606. 10366v1 Announce Type: cross Abstract: Simulation has become an essential tool for evaluating and improving vision-language-action (VLA) policies, offering scalable, reproducible, and controllable alternatives to costly real-world robot evaluation.

By Shuo Wang, Hanyuan Xu, Yingdong Hu, Fanqi Lin, Yang Gao

robotics fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 10

Spatiotemporal Graph Transformer for 3D Neighborhood Interaction and Quality Prediction in Metal Additive Manufacturing

arXiv:2606. 10227v1 Announce Type: new Abstract: Metal additive manufacturing enables the fabrication of complex parts, but achieving consistent build quality remains challenging due to interactions induced by repeated layer-wise melting, solidification, and reheating across the 3D build.

By Joyce Karen Pelaez, Siqi Zhang, Hoo Sang Ko

llms multimodal

arXiv AI

Jun 10

A History-Aware Visually Grounded Critic for Computer Use Agents

arXiv:2606. 11078v1 Announce Type: new Abstract: Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments.

By Jaewoo Lee, Zaid Khan, Archiki Prasad, Justin Chih-Yao Chen, Supriyo Chakraborty, Kartik Balasubramaniam, Sambit Sahu, Elias Stengel-Eskin, Hyunji Lee, Mohit Bansal

llms agents multimodal benchmarks

arXiv AI

Jun 10

Density Ridge Selective Prediction for LLM and VLM Hallucination Detection under Calibration Label Scarcity

arXiv:2606. 10198v1 Announce Type: cross Abstract: Hallucination detection in large language and vision-language models is increasingly framed as selective prediction, where a detector assigns a confidence score and abstains when confidence is low.

By Nina I. Shamsi

llms multimodal benchmarks safety

arXiv AI

Jun 10

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

arXiv:2606. 10029v1 Announce Type: cross Abstract: Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream.

By Nikita Koriagin, Georgii Aparin, Nikita Balagansky, Daniil Gavrilov

llms diffusion multimodal safety

arXiv AI

Jun 10

Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction

arXiv:2606. 10571v1 Announce Type: cross Abstract: Adversarial examples reveal vulnerabilities in Vision-Language Pre-training (VLP) models and provide insights for improving robustness.

By Lijia Yu, Jiuxin Cao, Yuchen Qiang, Changhao Chen, Yifei Huang, Bo Liu

llms multimodal safety

arXiv Machine Learning

Jun 10

Multimodal Brain Tumour Classification Using Feature Fusion

arXiv:2606. 11107v1 Announce Type: cross Abstract: Clinicians diagnose brain tumors by synthesizing patient symptoms, medical history, and quantitative imaging data from modalities such as MRI and CT scans into a unified clinical judgement.

By Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan, Volker Steuber

multimodal

arXiv AI

Jun 10

LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts

arXiv:2606. 09907v1 Announce Type: cross Abstract: Multimodal clinical learning is increasingly important for integrating diverse patient data, including imaging, text, and personalised health records.

By Maxx Richard Rahman, Prakhar Kumar, Wolfgang Maass

nlp multimodal

arXiv AI

Jun 10

Sigma-Branch: Hierarchical Single-Path Network Reconstruction for Dynamic Inference with Reduced Active Parameters

arXiv:2606. 09924v1 Announce Type: cross Abstract: Deploying deep neural networks on memory-constrained edge accelerators is bottlenecked by per-inference off-chip weight transfer rather than computation: the dense network cannot be retained on-chip, and every parameter must be loaded for every input.

By Kohga Tanaka, Hiroaki Nishi

fine-tuning efficiency multimodal

arXiv AI

Jun 10

A Unified Multi-Modal Framework for Intelligent Financial Systems: Integrating Reinforcement Learning, High-Frequency Trading, and Game-Theoretic Approaches with Cross-Modal Sentiment Analysis

arXiv:2606. 10412v1 Announce Type: new Abstract: The rapid evolution of financial technology demands sophisticated artificial intelligence systems capable of handling diverse challenges across multiple domains simultaneously.

By Fanrong Liu, Zhang Yuwei, Mingni Luo

llms rag nlp reinforcement-learning multimodal benchmarks

Get the daily briefing