Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2608.21714v1 Announce Type: new Abstract: Recent page-image retrievers such as ColPali have improved retrieval over visually rich documents, yet little is known about how they behave in cross-l...
arXiv:2608.21808v1 Announce Type: new Abstract: Multimodal Retrieval-Augmented Generation (RAG) with visual citation is crucial for ensuring the traceability and verifiability of MLLMs. However, curr...
arXiv:2608.21880v1 Announce Type: new Abstract: Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely wri...
arXiv:2608.23234v1 Announce Type: new Abstract: In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language M...
The paper introduces a framework that separates physical modeling from execution in physics reasoning tasks. It uses a two‑stage post‑training approach: supervised fine‑tuning to build structured models and reinforcement learning with rubric‑based feedback to refine them. Experiments on PhysReason, PhyX, and SeePhys show that this explicit modeling improves reasoning performance by about 3% on average for small LLMs.
The Cultural Moment Benchmark (CMB) evaluates video cultural reasoning in Southeast Asia by testing three distinct abilities: naming a cultural concept, visually recognizing it in a video, and locating its sub‑events in time. It contains 306 expert‑curated concepts from seven countries across five categories, with each concept assessed through three stages that use semantic‑similarity distractors, unlabeled video moments, and free‑form temporal localization. Experiments on six vision‑language models reveal varied failure modes, limited cascading between abilities, and differing impacts of audio and subtitles, while a human study shows even experts struggle with concepts from neighboring countries.
The paper introduces TimeCatch, a benchmark that evaluates temporal consistency in vision‑language models (VLMs) by treating temporal grounding as an anomaly detection problem. Temporal anomalies are created by swapping consecutive frames, while frame‑level anomalies involve replacing a frame with Gaussian noise. Across synthetic and real‑world datasets, VLMs reliably detect and localize frame‑level anomalies but perform near chance on temporal anomaly detection, whereas humans excel at both tasks.
SketchFlow is a new generative framework for creating high‑quality vector sketches from text prompts. It uses a Gaussian Mixture Model prior in the CLIP latent space and an Optimal Transport Conditional Flow Matching model to map this prior to sketch features, which are then decoded by a Hybrid Diffusion Decoder combining 1D U‑Net and Transformer architectures. The approach achieves superior visual quality and human‑like drawing styles, and supports zero‑shot synthesis for unseen concepts and smooth semantic interpolation.
SynerMedGen is a unified framework that aligns medical multimodal understanding with generation tasks through task alignment. It introduces three generation‑aligned understanding tasks and a two‑stage training strategy that transfers representations learned during understanding to medical image synthesis. The model achieves strong zero‑shot performance on 22 synthesis tasks and outperforms state‑of‑the‑art specialized and unified models when combined with generation training, supported by a new 1M‑sample SynerMed dataset.
The paper introduces MemeMind, a large-scale dataset for detecting harmful memes that includes a detailed taxonomy and Chain-of-Thought reasoning annotations. It also proposes MemeGuard, a multimodal framework that uses a three-stage training strategy to improve visual understanding, reasoning, and discrimination of harmful content. Experiments show MemeGuard surpasses current state-of-the-art methods on MemeMind, advancing detection accuracy and interpretability.
An end-to-end-trained vision-language model generates prostate biopsy reports in native languages, demonstrated in German. The system uses a tokenizer and model trained from scratch and an automated pipeline that splits composite reports into image-text pairs, producing 17,344 pairs from 2,402 cases without manual annotation. Evaluated on clinical attributes, it achieves 96.2% F1 for malignancy detection and 65.2% for Gleason grading, comparable to an FDA-cleared classifier and validated on external cohorts.
arXiv:2608.21868v1 Announce Type: new Abstract: Depression assessment from multimodal clinical interviews requires integrating dispersed evidence from multiple symptoms into a coherent PHQ-8 profile....
arXiv:2608.05154v3 Announce Type: replace Abstract: Multimodal rotary positional encodings apply temporal, height, and width phases to interleaved text, image, and video tokens. This creates two ambi...
The paper introduces SRAIN, a framework that learns sample‑wise, rank‑aware interpolation weights for composed visual data retrieval. Instead of relying on complex multimodal large language models, SRAIN uses simple linear interpolation in embedding space, dynamically predicting query‑specific weights through batch‑wise rank‑aware estimation and a compact memory bank for hard negatives. This approach achieves state‑of‑the‑art performance on composed video retrieval and competitive results on composed image retrieval while significantly reducing query‑time latency.
The paper introduces VIG (Visual Information Gain), an information‑theoretic reward that evaluates each token in a multimodal chain‑of‑thought by measuring how much the image reduces its predictive uncertainty. VIG is computed online using two forward passes—one with and one without the image—eliminating the need for reference chains or external annotations. Experiments on six multimodal reasoning benchmarks and multiple Qwen3‑VL‑Thinking model sizes show that VIG consistently improves the accuracy–efficiency trade‑off, demonstrating that efficient multimodal reasoning arises from increasing visual information density rather than merely limiting chain length.
arXiv:2511.05782v3 Announce Type: replace Abstract: Unsupervised domain adaptation (UDA) for medical image segmentation remains challenging due to substantial domain shifts across imaging modalities,...
The paper explores remaining useful life (RUL) estimation using multimodal large language models (MLLMs) that are grounded through time‑series retrieval. It proposes a framework that retrieves historically similar degradation segments, combines them with the test trajectory into a visual comparison artifact, and processes this via a structured multimodal prompt. Experiments on the FD001 partition of the C‑MAPSS benchmark show that retrieval consistently improves RUL prediction, with greater benefits for larger MLLMs, while also revealing current limitations in practical prognostics and health management (PHM) settings.
arXiv:2512.01946v4 Announce Type: replace-cross Abstract: Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in r...
arXiv:2608.21595v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts wi...
arXiv:2608.23435v1 Announce Type: cross Abstract: Understanding a basketball game requires recognizing events, localizing actions, identifying players, and relating these to structured game knowledge...