Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2608. 20019v1 Announce Type: new Abstract: Incomplete multimodal sentiment analysis has garnered significant attention in recent years.
arXiv:2608. 02092v2 Announce Type: replace Abstract: Deep multimodal fusion for object detection has demonstrated good performance through mining modal characteristics.
arXiv:2608. 19536v1 Announce Type: cross Abstract: Learning-based global point cloud registration has achieved remarkable progress, yet its reliance on geometric representations makes existing methods sensitive to variations in point density, scan pattern, viewpoint, and sensor characteristics.
arXiv:2608. 19729v1 Announce Type: new Abstract: Vision-language-model-based embodied agents can complete instructed tasks but often violate safety constraints in the process, a problem recently framed as interactive safety.
arXiv:2602. 07104v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have advanced the capabilities to interpret and act on visual input in 3D environments, empowering diverse applications such as robotics and situated conversational agents.
Personalized interpretation of medical reports has emerged as an increasingly important need among patients. Addressing this need requires both evidence-grounded medical factuality and context-dependent patient communication, yet existing medical vision-language tasks do not adequately capture these dual requirements.
The paper introduces V-REX, a specialist vision‑language model (VLM) tailored for veterinary X‑ray interpretation. By redesigning the VLM pipeline—including text tokenisation, pre‑training, grounding, and inference—the authors demonstrate that a carefully engineered model can outperform larger generalist foundation models while using far fewer parameters, data, and compute. V-REX is the first VLM capable of generating diagnostic reports for veterinary radiographs, achieving significant gains over open foundation models on this task.
Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention.
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions.
Multimodal LLMs can see a document, but they often can't read it reliably. Small text, tables, visual cues, and topological elements still trip them up under direct visual inference, even when the pag...
Large vision-language models (LVLMs) have achieved remarkable progress in video understanding and reasoning. Despite extensive studies on text- and image-based jailbreaks, video jailbreaks against LVL...
Vision-language-model-based embodied agents can complete instructed tasks but often violate safety constraints in the process, a problem recently framed as interactive safety. Training such agents to act safely is difficult, since safety and task success are distinct objectives, and safety arises only at a small number of safety-critical steps within a trajectory.
Latent reasoning has advanced multimodal reasoning through a two-stage training paradigm: (1) a helper image is encoded into latent tokens to teach visual chain-of-thought during a supervised fine-tuning (SFT) stage, and (2) these latent tokens are further refined with reward feedback during a reinforcement learning (RL) stage. In this paper, we identify two key limitations of this framework, one in each stage.
The paper investigates how Vision Language Models (VLMs) can be fooled by small, human‑imperceptible changes to images. It introduces a gradient‑based attack that targets only the vision encoder, reducing computational cost while still effectively disrupting both untargeted and targeted multimodal alignment. Experiments on open‑source VLMs such as Qwen2.5‑VL, Granite‑Vision, FastVLM, and Phi‑3.5‑Vision demonstrate that these perturbations can significantly alter the models’ textual outputs.
The paper introduces HN-CLIP, a new objective for dense-caption retrieval that adapts similarity margins per negative example using the text encoder’s own geometry. By adding a detached caption‑similarity matrix to the negative logits, HN‑CLIP addresses the issue of near‑duplicate captions that cause premature loss saturation in InfoNCE training. Experiments on four benchmarks show that HN‑CLIP outperforms leading methods by 2.4–4.3 R@1, trains 2.4× faster than GOAL and 5.4× faster than StructXLIP, and achieves the best full‑data baseline with only 20% of the training data.
The paper introduces a new benchmark for diagrammatic reasoning in olympiad geometry, comprising 954 self‑contained problems and a 297‑problem hard subset. Each problem is paired with a human‑authored, high‑fidelity diagram in Asymptote code and a suite of metrics for evaluating diagram construction. Experiments show that current foundation models excel at solving the problems but produce markedly less faithful diagrams, with an average compile success rate of only 36.14%.
The paper introduces BudgetDoc, a multimodal benchmark that explicitly supervises the trade‑off between inference budget and performance across three document tasks. Using this benchmark, the authors train DRB, a lightweight 1‑billion‑parameter pre‑flight estimator (SigLIP‑2 + Qwen3‑0.6B) that predicts ordinal model performance for different budget levels and achieves a weighted F1 of 0.753. When DRB dynamically allocates reasoning budgets to five frontier models on three datasets, it matches or improves F1 scores compared to always‑maximum‑budget baselines in 9 of 15 configurations while dramatically cutting cost, and preliminary tests suggest it may generalize to cross‑model selection.
The paper introduces a fine‑tuned Mistral‑7B‑instruct model as an automated "Supervisor‑in‑the‑Loop" for mental health care, using a tri‑stream analysis of therapeutic alliance, latent risk, and supervisory triage. Leveraging 106 DAIC‑WOZ sessions, the VAL (Visual‑Acoustic‑Linguistic) framework achieves high accuracy in technique identification, alliance assessment, and risk prediction, while cutting supervisory triage latency from 72 hours to about 10 seconds per session. The system also addresses cold‑start challenges with Bayesian priors and synchronizes modalities via timestamps for robust fusion.
The paper examines how multilingual medical adaptation affects the internal representations of Whisper ASR models. By comparing various fine‑tuning strategies—zero‑shot decoding, English‑only, German‑only, two‑stage EN→EN+DE, and direct EN+DE fine‑tuning—it shows that fine‑tuning significantly improves performance, with the best model varying by setting. Layer‑wise encoder analysis reveals that English medical fine‑tuning drives the main representation shift, while multilingual continuation largely preserves the adapted space, and that domain and language signals remain recoverable across layers.