Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2609. 09404v1 Announce Type: cross Abstract: Agentic AI frameworks let a language model plan, keep memory, and call tools that reach real files, mail, and services.
RiVaT‑Fuse introduces a reliability‑calibrated variational tensor fusion framework for multimodal image‑metadata prediction, treating fusion as a sample‑wise latent‑state estimation rather than simple aggregation. It replaces scalar modality confidence with matrix‑valued trust geometry, decomposes interactions into additive, multiplicative, and relational components, and couples the latent state with conditional robustness and structured multi‑task prediction. On an image‑level benchmark, RiVaT‑Fuse outperforms direct representation‑level baselines and improves probability and label stability under perturbation.
The paper critiques current optimization-based methods for balancing modalities in Multimodal Sentiment Analysis, arguing they overpromise and underdeliver. It introduces a unified evaluation framework that tests gradient- and loss-based balancing strategies, provides a theoretical diagnosis showing these methods conflate fitting speed with discriminative contribution, and proposes a research agenda for held‑out discriminative modality valuation. Experiments on CMU‑MOSI and CMU‑MOSEI demonstrate that no strategy consistently outperforms Late Concatenation, performance is highly sensitive to hyperparameters, and ratio calibration does not yield reliable gains, highlighting that loss is not utility and gradients are not importance.
MultiHuSE is a multimodal dataset featuring 2,407 high‑definition videos of 50 diverse actors delivering 1,463 text samples in four psychological humour styles—affiliative, aggressive, self‑enhancing, and self‑deprecating—plus neutral content. Each text is performed by multiple actors, allowing analysis of expressive diversity, and a subset includes emotion annotations. Baseline experiments show that multimodal fusion improves humour style classification accuracy over unimodal approaches, especially for affiliative humour.
The paper introduces methods to improve speech recognition for multilingual video transcription using Whisper-based tools, targeting cross‑cultural understanding. It reports an average transcription error rate of 30% across seven languages, which can be lowered to 20% with modest fine‑tuning. The authors also release associated speech and metadata to aid community refinement of these techniques.
The paper investigates whether Vision‑Language Models (VLMs) can understand visual persuasiveness by evaluating image‑message pairs that humans consistently judge as persuasive. It introduces Visual Persuasive Factors (VPFs), a taxonomy from cognitive psychology, to quantify visual cues influencing persuasive judgments. Empirical analysis shows VLMs tend to over‑predict persuasiveness, partially reproducing human patterns but often generating false positives, and that VPF‑guided interventions can improve performance only when properly framed.
The paper introduces a densely labeled coastal dataset of over 1,000 images from Oahu, Hawaii, featuring 18 semantic classes and 7,400 annotated instances. Seven modern vision‑language models were evaluated using text‑to‑mask, mask‑to‑mask, and mask‑to‑text alignment experiments. Results show that broad landscape classes are recognized more accurately than conventional object and coastal classes, with coastal concepts posing the greatest challenge; however, performance differences are not solely due to environmental context, and alternative textual labels improve recognition of several coastal concepts.
CLIP-RD introduces a relational distillation framework for efficient CLIP knowledge distillation, featuring Vertical Relational Distillation (VRD) and Cross Relational Distillation (XRD). VRD aligns intra‑modal similarity distributions between teacher and student, while XRD aligns cross‑modal similarity distributions to enforce bidirectional symmetry. This joint modeling of multidirectional relational structures improves the student’s embedding geometry, yielding a 1.8%p performance gain over CLIP‑KD across various architectures, tasks, and corruption settings with minimal training‑time overhead.
V‑Retrver is an evidence‑driven retrieval framework that treats universal multimodal retrieval as an agentic reasoning process grounded in visual inspection. It allows multimodal large language models to selectively acquire visual evidence through external tools, alternating between hypothesis generation and targeted visual verification. The approach is trained with a curriculum that blends supervised activation, rejection‑based refinement, and reinforcement learning, achieving an average 23.0% improvement in retrieval accuracy across multiple benchmarks.
ABACUS is a 3‑billion‑parameter vision‑language model that simultaneously tackles object counting, crowd counting, referring‑expression counting, and count‑faithful image generation. It introduces density‑aware adaptive zooming with an objectness map, a boundary‑aware count policy trained via GRPO to avoid over‑ or under‑counting at crop edges, and a cycle‑consistent GRPO strategy that scores generated images for count accuracy and aesthetic quality without external critics. The model sets new state‑of‑the‑art performance on seven benchmarks, outperforming both specialized and larger generalist models.
The paper reports an entry to the EgoLongQA track of the Wearable‑AI Challenge at ECCV 2026 that achieved first place in the ≤2 B parameter division with a score of 0.8279 on the held‑out test set. The system is a single 2 B vision‑language model that answers multiple‑choice questions about ten‑minute egocentric videos in one greedy forward pass, obtained by distilling the junior perception module of a tool‑using agentic pipeline into a small student using teacher traces that answered correctly. By pruning the multilingual embedding table from 248,320 to 143,469 rows, the authors reduce the model to 1.9985 B parameters while maintaining identical logits on retained rows, enabling the model to reach 89 % of the accuracy of the large agentic pipeline with only 1.1 % of its parameters. whyItMatters":"The work demonstrates that a highly efficient, sub‑2 B parameter vision‑language model can match the performance of a much larger agentic pipeline on long‑video perception tasks, showing the feasibility of deploying such models in resource‑constrained settings."
The paper introduces SurgicalRoomAgent, a voice‑interactive multi‑agent system for smart operating rooms that leverages large language models to understand natural language, control devices, record intraoperative events, and generate surgical reports. Its layered architecture includes a voice interaction pipeline (wake, ASR, turn detection, agent reasoning, TTS) and an agent core (skill registry, task planner, device manager). Three key technologies—KV Cache prefix warming, streaming partial JSON parsing with early parallel task execution, and progressive skill prompt disclosure—reduce latency and maximize context efficiency, enabling real‑time operation within a 16,384‑token limit.
The paper presents a lightweight vision‑language encoder paired with a compact multi‑layer perceptron to estimate flare combustion efficiency from low‑cost thermal video. The trained model is deployed via a user‑friendly GUI that overlays efficiency values on each frame, tracks real‑time trends, visualizes distribution across the video, and exports CSV reports. In a six‑month trial, the system maintained 99% uptime with under 15 minutes of weekly maintenance.
The paper investigates using Vision Language Models (VLMs) to accelerate verification and validation (V&V) of classification models by automatically detecting systematic errors. It introduces a VLM-based error slice detection (ESD) method that groups and labels errors, demonstrating its ability to identify perturbations in a non-military dataset and to cluster images by surroundings in a military context. The study highlights challenges such as underrepresentation of defence data in VLM training and limited contextual diversity, and suggests that while fully automated V&V is not yet feasible, VLMs could speed up the process in the future.
HALDETECT is a system developed for the English hallucination-detection track of ImageEval 2026, where the task is to identify the single visually grounded statement among three culturally plausible options. The approach treats the problem as a contrastive decision, outputs the answer before an explanation, and bases reasoning on colour/texture, shape/form, and context. The best model fine‑tunes Qwen2.5‑VL‑7B‑Instruct with 4‑bit QLoRA, freezes the vision encoder, and achieves a Contrastive Instability score of 0.035 on the test set, placing third among eight teams.
The paper announces VectraYX‑Vision‑1B, a Spanish/LATAM cybersecurity vision‑language model that uses a natively‑trained visual tower (Qwen2‑VL) and a frozen decoder. The authors report that transplanting the visual tower improves the failing 8‑nibble address field from 0.00 to 0.81 exact, using a coarser token budget than 2×2 tiling, and that a second pre‑registered field shows 63% contamination and is demoted. The model’s B6/B7 tool‑id remains at floor, and code/checkpoints are available on Hugging Face.
The paper presents a method for robust multimodal sentiment analysis that handles incomplete or noisy modalities. It introduces a completeness estimation technique to measure how much sentiment-relevant information remains in partial data, guiding the reconstruction of missing semantics. A joint training strategy stabilizes multi-task learning for sentiment prediction and completeness estimation, and experiments on three benchmark datasets show improved semantic reconstruction and sentiment accuracy.
The paper introduces IRS (Incongruity-Resolution Supervision), a framework that breaks humor understanding into three parts: identifying mismatches in a visual scene, creating coherent reinterpretations of those mismatches, and aligning these interpretations with human preferences. IRS uses structured reasoning traces to guide models from visual perception to humorous interpretation, and it is evaluated on the New Yorker Cartoon Caption Contest. Experiments on 7B, 32B, and 72B models show that IRS improves caption matching and ranking, with the 72B model achieving 76.10% ranking accuracy—outperforming non-expert humans and all other multimodal baselines—and demonstrates transferable reasoning patterns in zero‑shot settings.
Recursive Code World Models (RCWM) present a framework that reconstructs complex 3D worlds from a single reference image by combining a Recursive Scene Program (RSP) representation with a construction solver that recursively calls itself. The RSP encodes the world as compositional scene code, while each solver invocation follows a global‑local‑global recursion: it first establishes the whole scene, then recursively reconstructs unresolved parts, and finally revisits the whole to refine composition. This approach preserves scene‑wide geometry and relationships, propagates a shared camera projection across levels, and uses a vision‑language coding agent to compare reference images with scene renders, enabling fine‑scale structure refinement and improved reconstruction over prior methods.
The paper introduces HuRo, a dataset of 630K robotized episodes derived from diverse human videos, created via a pipeline that aligns observations and actions for robotic use. Experiments on four real‑world manipulation tasks show that scaling robotized pretraining boosts task completion from 51.5% to 80.3% and improves out‑of‑distribution performance under spatial and visual shifts. Ablation studies reveal that visual robotization enhances robustness and that end‑to‑end pretraining with retargeted actions outperforms visual‑only transfer.