Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,396 stories · RSS feed

arXiv AI

Jul 3

Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning

arXiv:2512. 14157v2 Announce Type: replace Abstract: Recent medical MLLMs have made significant progress in generating step-by-step textual reasoning chains.

By Yankai Jiang, Yujie Zhang, Peng Zhang, Wenjie Li, Yichen Li, Jintai Chen, Xiaoming Shi, Shihui Zhen

agents computer-vision reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 3

VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval

arXiv:2607. 02371v1 Announce Type: cross Abstract: Over 285 million people worldwide live with a visual impairment, for whom everyday tasks such as avoiding obstacles, locating personal belongings, recognizing familiar faces, or handling cash remain persistent obstacles to personal autonomy.

By Cristian-Gabriel Florea, Stelian Sp\^inu

llms rag computer-vision efficiency multimodal

arXiv AI

Jul 3

AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

arXiv:2607. 02269v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated immense promise in Spatio-Temporal Video Grounding (STVG).

By Rintaro Otsubo, Ryo Fujii, Reina Ishikawa, Taiki Kanaya, Kanta Sawafuji, Hiroki Kajita, Shigeki Sakai, Hideo Saito, Ryo Hachiuma

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 3

ESC: Emotional Self-Correction for Reliable Vision-Language Models

arXiv:2607. 02089v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, yet they remain vulnerable to unreliable reasoning.

By Tien-Huy Nguyen, Minh-Nhat Nguyen, Nguyen Nhat Huy, Hung Viet Nguyen, Huy Nguyen Minh Nhat, Thanh-Huy Nguyen, Cuong Tuan Nguyen, Hoang M. Le, Dat Nguyen, Phat Kim Huynh, Min Xu, Ulas Bagci

llms multimodal benchmarks safety

arXiv AI

Jul 3

An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

arXiv:2607. 02119v1 Announce Type: cross Abstract: While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation.

By Haoran Wang, Jinchuan Tian, Siddhant Arora, Shinji Watanabe

llms efficiency multimodal

arXiv AI

Jul 3

Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias

arXiv:2607. 01973v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly applied in medical tasks such as pathology description, report generation, and visual question answering.

By Sofiane Ouaari, Kevin Vorwalder, Nico Pfeifer

llms rag nlp multimodal benchmarks safety

arXiv AI

Jul 3

SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models

arXiv:2607. 01876v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding, yet their enormous parameter scale and cross-modal computation incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices.

By Qi Lyu, Jiahua Dong, Baichen Liu, Xudong Wang, Mingfei Han, Yulun Zhang, Fahad Shahbaz Khan, Salman Khan, Lianqing Liu, Zhi Han

llms multimodal

arXiv AI

Jul 3

ProCal: Inference-Time Proposal Calibration for Open-Vocabulary Object Detection

arXiv:2607. 01759v1 Announce Type: cross Abstract: Open-vocabulary object detection aims to localize and classify objects beyond the fixed set of categories seen dur ing training.

By Jae-Ryung Hong, Ho-Joong Kim, Seong-Whan Lee

rag computer-vision multimodal

arXiv AI

Jul 3

CPG-PAD: Concept-Informed Prompts Guided Presentation Attack Detection

arXiv:2607. 01303v1 Announce Type: cross Abstract: Presentation Attack Detection (PAD) serves as a crucial safeguard for face recognition systems against presentation attacks such as printed photos, replayed videos, and 3D masks.

By Haoyuan Zhang, Xiangyu Zhu, Li Gao, Ajian Liu, Siran Peng, Zhen Lei

llms multimodal benchmarks

arXiv AI

Jul 3

Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment

arXiv:2607. 01239v1 Announce Type: cross Abstract: Character-level perturbations bypass safety alignment in modern LLMs despite leaving prompts human-readable.

By Tung-Ling Li, Hongliang Liu, Yuhao Wu

llms nlp reinforcement-learning fine-tuning multimodal safety

arXiv AI

Jul 3

Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing

arXiv:2607. 01978v1 Announce Type: new Abstract: Online multimodal knowledge editing requires injecting a continual stream of visual-textual corrections into multimodal large language models (MLLMs) with bounded overhead and minimal disruption to unrelated behaviors.

By Siyuan Li, Youyuan Zhang, Ruitong Liu, Junxi Wang, Jing Li

llms multimodal benchmarks

arXiv AI

Jul 3

MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support

arXiv:2607. 01814v1 Announce Type: new Abstract: Traditional Chinese Medicine (TCM) diagnosis, particularly through tongue inspection, faces persistent challenges in subjectivity and reproducibility.

By Lihui Luo, Joongwon Chae, Ziyan Chen, Yang Liu, Siyi Cheng, Weihan Gao, Zelin Zeng, Xiaoming Yin, Samaneh Beheshti Kashi, Dongmei Yu, Lian Zhang, Jing Sui, Zeming Liang, Jiansong Ji, Peter E. Lobie, Peiwu Qin

llms rag computer-vision fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 3

NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

arXiv:2601. 01095v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored.

By Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, Gargi Chakraborty

llms multimodal benchmarks

arXiv Machine Learning

Jul 3

LIME: Learning Intent-aware Camera Motion from Egocentric Video

arXiv:2607. 02417v1 Announce Type: cross Abstract: Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent.

By Boyang Sun, Jiajie Li, Yung-Hsu Yang, Chenyangguang Zhang, Tim Engelbracht, Sunghwan Hong, Cesar Cadena, Marc Pollefeys, Hermann Blum

agents robotics multimodal

arXiv AI

Jul 3

QFedAgent: Quantum-Enhanced Personalized Federated Learning for Multi-Agent Activity Recognition

arXiv:2607. 02426v1 Announce Type: cross Abstract: Federated learning (FL) enables collaborative model training across distributed devices without sharing raw data, making it suitable for privacy-sensitive robotic sensing applications.

By Quoc Bao Phan, Tuy Tan Nguyen

agents robotics multimodal safety

arXiv AI

Jul 3

BaRA: Budget-constrained and Reliable Web Data Collection Agent

arXiv:2607. 00007v2 Announce Type: replace-cross Abstract: Large language model (LLM)-based web agents automate web navigation and data collection.

By Soojeong Lee, Joseph Lee, Yongseong Cho, Sunjae Kim, Youngwoo Moon, Kyungwoo Song

llms agents multimodal

arXiv AI

Jul 3

VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment

arXiv:2607. 01586v1 Announce Type: cross Abstract: Vision-language-action models (VLAs) have recently advanced robotic manipulation, yet the effects of different robot-data pre-training paradigms remain difficult to compare because existing models often differ in architecture, data, action space, and evaluation protocol.

By Guoyang Xia, Fengfa Li, Hongjin Ji, Lei Ren, Fangxiang Feng, Kun Zhan, Yan Xie

robotics multimodal benchmarks safety

arXiv AI

Jul 3

SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings

arXiv:2607. 01238v1 Announce Type: cross Abstract: Recent advances in speech synthesis have shifted from phoneme representations to direct grapheme modeling.

By Priyam Mazumdar, Yurii Halychanskyi, Steven Guo, Mark Hasegawa-Johnson, Volodymyr Kindratenko

rag multimodal

arXiv AI

Jul 3

Multi-modal Rail Crossing Safety Analysis

arXiv:2607. 01365v1 Announce Type: cross Abstract: Given one or more images of a railway crossing, can we leverage visual cues that allow us to robustly estimate how safe it is?

By Paimon Goulart, Chansong Lim, N\'icolas Roque dos Santos, Yue Dong, Sheldon Peterson, Jia Chen, Evangelos E. Papalexakis

fine-tuning multimodal safety

arXiv AI

Jul 3

Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

arXiv:2607. 02484v1 Announce Type: cross Abstract: Visual token pruning is a crucial strategy for accelerating VLMs by compressing redundant image patches, yet existing methods often fail to preserve critical cues under dense instructions and fine-grained queries.

By Xuehui Wang, Xuankun Yang, Wei Shen

efficiency multimodal benchmarks

Get the daily briefing