Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,891 stories · RSS feed

arXiv AI

Aug 5

Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Execution

arXiv:2608. 03483v1 Announce Type: cross Abstract: Existing chunk-based Vision-Language-Action (VLA) models execute a fixed number of actions (i.

By Weichen Xu, Zhenhua Liu, Lin Luo, Yaobo Liang, Chengtang Yao, Qingyu Mei, Jian Cao, Xixin Cao, Xing Zhang, Jiaolong Yang, Baining Guo

reinforcement-learning robotics multimodal safety

arXiv AI

Aug 5

Standalone DINOv3 for Training-Free Open-Vocabulary Semantic Segmentation in Remote Sensing

arXiv:2608. 03023v1 Announce Type: cross Abstract: Remote sensing semantic segmentation is hindered by costly pixel-level annotations, motivating training-free open-vocabulary methods.

By Changhao Zhao, Haoxiang Li, Yuke Li, Hai Liu, LingLin Zeng

computer-vision fine-tuning multimodal

arXiv AI

Aug 5

UniGD: A Unified Generative-Discriminative Framework for Industrial Retrieval

arXiv:2608. 03150v1 Announce Type: new Abstract: Generative retrieval (GR) is a promising paradigm for industrial search advertising, yet its deployment is constrained by strict relevance and latency requirements.

By Shujie Ji, Yawei Kong, Yilin Zhao, Li Wang, Xialong Liu, Peng Jiang

efficiency multimodal

arXiv AI

Aug 5

CorePath: A Breast-Specialized Pathology Foundation Model for Core Needle Biopsy Diagnosis and Risk-Controlled Report Generation

arXiv:2608. 03079v1 Announce Type: cross Abstract: Breast core needle biopsy (CNB) is central to breast cancer diagnosis yet remains challenging because limited tissue sampling, lesion heterogeneity, and subtle morphologic overlap can obscure subtype distinctions.

By Ting Yin, Danning Li, Chen Shu, Xiaoxia Yao, Boyu Fu, Yujing Chang, Tianyu Shi, Mengna Feng, Jie Chen, Jing Fu, Xiuli Xiao, Tianlin Li, Mumin Shao, Jiaxin Bi, Wenchuan Zhang, Xiaoyan Wu, Xiao Han, Zhang Zhang, Yuhao Yi, Hong Bu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Aug 5

SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

arXiv:2607. 27703v2 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning.

By Yang Zhou, Zixuan Huang, Sunzhu Li, Zhuo Yang, Chen Zhang, Shunian Chen, Caijun Yan, Jianyao Xu, Shunyu Liu, Weijie Fu, Peiliang Li, Xiaozhi Chen, Yuxiang Cai

llms agents computer-vision reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Aug 5

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

arXiv:2608. 03450v1 Announce Type: cross Abstract: Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction.

By Haoqian Kang, Liupeng Li, Kuofeng Gao, Jinpeng Wang, Zhenyu Lu, Bin Chen, Ke Chen, Yaowei Wang

llms multimodal benchmarks safety

arXiv AI

Aug 5

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

arXiv:2608. 03979v1 Announce Type: cross Abstract: We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration.

By Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, Feng Zhao

llms agents fine-tuning multimodal benchmarks safety

arXiv AI

Aug 5

DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

arXiv:2608. 03451v1 Announce Type: new Abstract: Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia.

By Boyan Li, Zhuowen Liang, Yupeng Xie, Xiaotian Lin, Tianqi Luo, Xinyu Liu, Yizhang Zhu, Zhangyang Peng, Yuan Li, Zhengxuan Zhang, Jiayi Zhang, Nan Tang, Guoliang Li, Yuyu Luo

agents multimodal benchmarks safety

arXiv AI

Aug 5

Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping

arXiv:2608. 03050v1 Announce Type: cross Abstract: What is music style?

By Jingwei Zhao, Gus Xia, Ziyu Wang, Ye Wang

llms multimodal safety

arXiv AI

Aug 5

Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents

arXiv:2608. 03327v1 Announce Type: new Abstract: Hybrid computer-use agents can act through screenshots or call text tools.

By Siqi Fan, Minghao Li, Xiaoqian Ma, Wenhui Tan, Xiusheng Huang, Juntong Wu, Liujie Zhang, Shuo Shang, Weihang Chen

agents reinforcement-learning multimodal benchmarks

arXiv AI

Aug 5

Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages

arXiv:2508. 05149v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated potential in handling spoken inputs for high-resource languages, reaching state-of-the-art performance in various tasks.

By Seraphina Fong, Marco Matassoni, Alessio Brutti

llms multimodal benchmarks

arXiv AI

Aug 5

Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

arXiv:2608. 03733v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate.

By Chunyang Jiang, Pingping Zhang, Yuzhi Zhao, Wenao Ma, Zhijian Hou, Mengyang Wu, Yiyang Cai, Senkang Hu, Sitong Cheng, Chi-Min Chan, Wei Xue, Yike Guo

llms nlp multimodal benchmarks

arXiv AI

Aug 5

Learning Molecular Representations from Cellular Phenotypes with Structure Preservation

arXiv:2608. 02688v1 Announce Type: cross Abstract: Phenotypic drug discovery enables the discovery of functional relationships between molecular structures and cellular responses.

By Xuan Lin, Jingyu Sheng, Tengfei Ma, Li Sun, Dapeng Xiong

rag multimodal safety

arXiv AI

Aug 5

ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies

arXiv:2608. 02958v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies trained by behavior cloning fail silently: from the action stream alone, a collapsing rollout looks much like one making clean progress, because imitation supplies no notion of progress.

By Inkyu Sa, Konstantin Stulov, Rajat Bhageria

llms agents reinforcement-learning robotics multimodal

arXiv AI

Aug 5

OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet

arXiv:2608. 03428v1 Announce Type: cross Abstract: Image based dietary assessment offers a scalable alternative to self reported food diaries, yet fine-grained food recognition remains challenging due to high intra-class variability and visually similar dishes.

By Dimitrios I. Zaridis, Traianos Tsiokris, Vasileios C. Pezoulas, Daphni Plati, Eugenia Mylona, Eleni Georga, Nikos Tsiknakis, Antonis Sakellarios, Dimitrios I. Fotiadis

llms fine-tuning multimodal

arXiv AI

Aug 5

UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space

arXiv:2608. 03817v1 Announce Type: cross Abstract: Large vision--language models (LVLMs) demonstrate strong multimodal reasoning capabilities but remain prone to hallucination, where model predictions are not grounded in visual evidence.

By Amir Mohammad Ezzati, Kiyan Rezaee, Bardiya Kariminia, Mohamad Amin Yousefi, Asal Mohammadjafari Mamaqani, Behrad Samimi, Mohammad Hossein Rohban

llms multimodal benchmarks safety

arXiv AI

Aug 5

In-Context Collapse in Vision-Language Models and How to Mitigate it?

arXiv:2608. 02830v1 Announce Type: cross Abstract: Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied.

By Mohammad Rostami

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Aug 5

DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack

arXiv:2608. 03207v1 Announce Type: cross Abstract: Flow-matching vision-language-action (VLA) models such as pi0 generate robot actions by integrating a learned denoising velocity field, and have been reported to resist adversarial perturbations that readily fool autoregressive VLAs.

By Hoseong Tae, Jong-Seok Lee

rag robotics multimodal safety

arXiv AI

Aug 5

When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

arXiv:2608. 03918v1 Announce Type: cross Abstract: Efficient long-video understanding requires vision--language models (VLMs) to reason over a small number of frames selected as sparse visual evidence.

By Ke Li, Jiayu Chen, Maoliang Li, Zihao Zheng, Hailong Zou, Hengyi Zhang, Xuanzhe Liu, Xiang Chen

llms agents multimodal

arXiv Machine Learning

Aug 5

DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

arXiv:2608. 02769v1 Announce Type: cross Abstract: Multimodal supervised learning seeks to leverage multiple heterogeneous data sources to improve predictive performance.

By Sagnik Nandy, Samriddha Lahiry, Pragya Sur, Subhabrata Sen

rag multimodal benchmarks