Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,971 stories · RSS feed

arXiv Computation and Language
Sep 25

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

SemMSA introduces a latent semantic‑aided framework for multimodal sentiment analysis that leverages large language models to generate rich sentiment‑relevant semantics. The method employs Cross‑modal Semantic Refinement (CSR) to fuse visual, acoustic, and language features in a frozen LLM embedding space, and Cross‑modal Spectral Alignment (CSA) to align these refined semantics with all modalities via spectral enhancement of kernel Gram matrices. Experiments on SIMS, MOSI, and MOSEI benchmarks show that SemMSA achieves state‑of‑the‑art performance.

By Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang
arXiv Computer Vision
Sep 25

OmniFabric: Coherent UV Space Texture Synthesis for 3D Garment Reconstruction

OmniFabric is a new method for creating production‑ready 3D garment assets from a single image. It generates globally coherent texture maps directly in the 2D sewing pattern (UV) space, using a coarse initialization from Vision‑Language Models and refining it with a diffusion transformer conditioned on 3D positional features. The approach removes distortion and baked‑in artifacts, producing photorealistic 3D garments with high‑quality textures that outperform current state‑of‑the‑art baselines.

By Ding-Jiun Huang, Yuanhao Wang, Cheng Zhang, Hugo Bertiche, Alexandru-Eugen Ichim, Thabo Beeler, Fernando De la Torre
arXiv Machine Learning
Sep 25

ICE: Task-Aligned Clifford Latent Fields for Multimodal Graph Foundation Models

ICE (Interaction-aware Clifford Encoder) is a multimodal graph foundation model that uses a node-indexed Clifford latent field to encode topology, text, and images into explicit Cl(3) addresses. Edge-aware geometric products transform these directions into scalar, bivector, and trivector relations over observed neighborhoods, preserving entity semantics while enabling higher-order transport and direct field access. Across eleven graphs and multiple node‑classification, link‑prediction, and few‑shot tasks, ICE outperforms all 30 reported supervised and few‑shot comparisons, with core removals and mechanism controls demonstrating the importance of its higher‑order structure and semantic protection.

By Xunkai Li, Xu Wang, Yinlin Zhu, Xiong Yongfu, Yi Liu, Rong-Hua Li, Guoren Wang
arXiv Computer Vision
Sep 25

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

CinematicVQA is a new benchmark for evaluating large vision‑language models on film‑grammar reasoning. It introduces the Cinematic Scene Graph, a structured representation linking filming techniques to perceptual effects and narrative functions, and tests models on tasks beyond low‑level technique recognition. The study finds a semantic gap where models excel at describing visuals but struggle to identify underlying techniques, and shows that fine‑tuning improves performance on narrative function and multi‑hop reasoning.

By Shuo Xing, Pooja Verlani, Balu Adsumilli, Zhengzhong Tu
arXiv AI
Sep 25

DrGait: Biomechanically Grounded Visual Reasoning for Interpretable Clinical Gait Analysis

DrGait is a training‑free framework that transforms Vision‑Language Models into clinical planners for gait analysis. It separates semantic reasoning from geometric perception using a Triage‑Verification‑Synthesis workflow, where hypotheses are generated, verified with deterministic biomechanical tools, and refined in a closed‑loop. This approach reduces hallucinations and produces transparent, audit‑ready clinical reports with competitive diagnostic accuracy.

By Xiangyu Yin, Shiqi Wang, Abrar Alamri, Yasir Aljohani, Weichen Liu, Goeran Fiedler, Wei Gao
arXiv Computer Vision
Sep 25

Direction-Scale Decomposition in Action Representation: Rethinking What to Tokenize for Vision-Language-Action Models

The paper introduces Direction-Scale Decomposition (DSD), an action representation that separates translation and rotation increments into direction and scale components before tokenization. DSD is evaluated with uniform binning and a B-spline tokenizer (BEAST) in both simulation and real-world manipulation tasks, showing improved success rates on LIBERO and SimplerEnv, especially under mixed-dataset training. Real-robot experiments confirm performance gains with and without robotics pretraining, supporting DSD as an effective representation for discrete-token vision-language-action models.

By Yufei Duan, Hang Yin, Alberta Longhini, Chao Tang, Danica Kragic
arXiv Computation and Language
Sep 25

Persona Prompting in Multimodal Urban Perception: Descriptive Convergence and Interpretive Variation

The paper investigates how persona prompting influences the language produced by two multimodal large language models—Qwen3‑VL and Gemma4—when describing urban scenes. Outputs are categorized into descriptive grounding (captions), perception tags, and interpretive framing (justifications). Results show captions largely converge across persona profiles, while justifications differ markedly, especially along economic status, political orientation, and personality dimensions, with economic status producing the greatest variation. Perception tags also reflect attribute similarities, and exploratory topic analysis indicates persona‑specific evaluative emphasis. Overall, persona prompting has a stronger effect on interpretive framing than on descriptive grounding.

By Neemias da Silva, Matt Ratto, Myriam Delgado, Rodrigo Minetto, Daniel Silver, Thiago H Silva
arXiv Computer Vision
Sep 25

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

RGBD20K is a new large-scale RGB‑D semantic segmentation dataset featuring 20,000 image pairs and 160 fine‑grained categories, surpassing existing benchmarks like NYUv2 and SUN RGB‑D in both scale and semantic diversity. The dataset provides high‑fidelity annotations obtained through rigorous re‑evaluation and correction of prior labels, ensuring a clean ground‑truth foundation. Additionally, the authors introduce a score‑purified fusion (SPF) method that achieves state‑of‑the‑art performance across evaluated benchmarks, demonstrating the value of high‑quality multimodal information.

By Shaohua Dong, Zexuan Meng, Haiyan Sun, Bing Fan, Cuicui Zhang, Dylan Joseph, Kewei Sha, Yunhe Feng, Heng Fan
arXiv AI
Sep 25

Planning Takes More Than Token Prediction: Causal Plan for Benchmarking and Building Physically Grounded Embodied Reasoners

The paper argues that current embodied vision‑language planning benchmarks favor linguistic next‑token prediction over physically grounded next‑state reasoning, leading models to rely on language priors rather than true causal dependencies. To address this, the authors introduce Causal‑Plan‑Bench, a diagnostic suite covering four causal dimensions, and Causal‑Plan‑1M, a million‑scale corpus of explicit causal reasoning traces extracted from egocentric videos. Extensive experiments show that existing models perform poorly on these tasks, while a new model trained with a tailored recipe—Causal Planner based on Qwen3‑VL‑8B—achieves significant gains, demonstrating the feasibility of physically grounded causal reasoning.

By Zheng Lu, Mingqi Gao, Qinlei Xie, Wanqi Zhong, Hanwen Cui, Zirui Song, Lijie Wang, Chong Luo, Bei Liu, Yiming Li
arXiv Computer Vision
Sep 25

Long-Tail Adaptive Flow Matching with Explicit Conditional Consistency Guidance for Precise Multimodal Face Synthesis

The paper introduces EC²Face, a multimodal face synthesis framework that enhances semantic alignment by combining Explicit Conditional Consistency Guidance (ECCG) and Long‑Tail Adaptive Flow Matching (LAFM). ECCG enforces pixel‑level consistency between generated faces, textual descriptions, and semantic masks, while a temporal dynamic modulation adjusts supervision strength over diffusion timesteps. LAFM reweights spatial optimization signals according to attribute frequency, improving rare attribute synthesis without adding inference overhead. Experiments demonstrate that EC²Face outperforms baselines, achieving a 29.38% improvement in mask accuracy for rare attributes.

By Yushe Cao, Xuechao Zou, Xing Xi, Dianxi Shi, Chun Yu, Junliang Xing
arXiv Computer Vision
Sep 25

GeoNLI - A Natural Language Interpreter for Satellite Imagery

GeoNLI introduces a unified, modular pipeline that combines advanced SAM variants with multimodal large language models to perform satellite image captioning, visual question answering (VQA), and visual grounding. The EarthMind model achieves strong results on captioning and VQA, while multiple RemoteSAM-SAM and DiffuSAM pipelines are used for grounding, ultimately employing a majority‑voting ensemble across several models. The system reports 82% captioning accuracy, 83.32% VQA accuracy, and 64.94% grounding accuracy, demonstrating improved consistency over task‑specific approaches.

By Ashutosh Gandhe, Anupam Rawat, Geet Sethi, Kabir Nasiruddin, Madhav Kotecha, Panav Shah, Rakshit Sawarn, Soumitra Nayak
arXiv Computer Vision
Sep 25

FoCal: Frequency-Oriented Cross-Modal Interaction and Spectral Calibration for Aerial Visible-Infrared Object Detection

FoCal is a new frequency‑oriented framework for aerial RGB–IR object detection that explicitly models cross‑modal interaction across different frequency components. It introduces a Frequency‑Aware Dual‑Domain Calibration module to consolidate low‑frequency structural cues while preserving high‑frequency modality‑specific details, and a Discrepancy‑Guided Spectral Modulation module that adaptively enhances, preserves, or attenuates the joint spectrum based on confidence‑weighted amplitude discrepancies. Experiments on DroneVehicle, ESCVehicle, and ATR‑UMOD show FoCal achieving high mAP scores (83.5%, 54.8%, 64.6%) with only 3.0 M parameters and 113.6 FPS, demonstrating a strong accuracy–efficiency trade‑off.

By Ben Liang, Chao Sui, Junqi Bai, Yuan Liu, Chunlai Li, Xiubao Sui, Qian Chen
arXiv Computer Vision
Sep 25

Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition

The paper introduces Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition, addressing head‑pose variation that causes appearance changes in VSR. It proposes a Dynamic Residual FiLM (DR‑FiLM) modulator that predicts input‑dependent weights to control pose‑conditioned modulation strength. Experiments on LRS2 and LRS3 show that DR‑FiLM reduces phoneme error rates compared to unweighted multi‑pathway modulation and that deeper FiLM pathways receive higher weights as head‑pose variation increases.

By Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh
arXiv Computer Vision
Sep 25

PROVE: Proof-guided Regime-aware Operator Verification for Hallucination Detection in Medical Visual Question Answering

PROVE is a black‑box hallucination detector for medical visual question answering that tailors its verification strategy to each question’s evidential structure. It classifies questions into three regimes, activates a subset of five operators per regime, and calibrates operator importance using deterministic question‑answer features to produce a risk score. On 8048 test samples across three medical VQA benchmarks and four state‑of‑the‑art vision‑language models, PROVE achieves an AUROC of 0.821, surpassing the best baseline by 0.159 with consistent improvements across all models and datasets.

By Keyang Zhou, Siyi Li, Zhongnan Shi, Qichao Ying, Wei Tang, Zhenxing Qian
arXiv Machine Learning
Sep 25

Exploiting answer-invariant redundancies in satellite imagery for efficient VLM inference on edge

The paper introduces Rift, a two‑stage system that reduces the computational load of vision‑language models on satellites by pruning image tiles that do not affect the answer and then applying elastic prefill to limit token usage. By exploiting answer‑invariant token redundancy, Rift cuts energy consumption by 78 % and latency by 69 % compared to exhaustive tiled inference, while boosting accuracy from 45 % to 73 % on LLaVA‑1.5 7B running on a Jetson AGX Orin.

By Ishani Janveja, Davis Zhang, Seoyul Oh, Deepak Vasisht
arXiv AI
Sep 25

C3M: Cross-Session Multimodal Memory Maintenance for Long-Horizon Tasks

C3M is a cross‑session multimodal memory system designed for long‑horizon tasks that must preserve and retrieve evidence across sessions within a limited, query‑blind memory budget. It maintains a bounded active index of source text‑image evidence, using relation‑aware updates to keep safe redundancy while preserving complementary and incompatible records. At query time, budgeted routing selects useful index pages and expands their associated source evidence under a fixed reader budget, creating a compact, provenance‑preserving memory that retains temporal distinctions and source links for reliable downstream reasoning.

By Xueshu Chen, Yan Wang, Zihao Xue, Jiefu Li, Zhenfang Liu, Jayden Chen, Zhen Bi, Jungang Lou
arXiv AI
Sep 25

SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection

SARFusion introduces a scene-aware routing approach for camera‑LiDAR 3D object detection, decoupling object‑query decoding into separate camera, LiDAR, and fusion branches. By estimating a global scene reliability prior and incorporating object‑level evidence, each query is routed to the most suitable branch, reducing cross‑modal interference. The method achieves strong performance on the nuScenes test set (72.5 mAP, 74.4 NDS) and demonstrates robustness to sensor corruptions and environmental changes.

By Yuting Zhao, Ziyi Zheng, Shuxiao Li
arXiv AI
Sep 25

Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models

The paper introduces Domain Recentering with Confidence Calibration (DRC), a training‑free technique that adapts CLIP to unlabeled target images by fitting a Gaussian mixture and subtracting a posterior‑weighted average of component means from each embedding. It further corrects residual class bias using a log‑prior adjustment based on confidence‑weighted predictions. DRC outperforms other methods, raising average accuracy on cross‑domain datasets by 4.13 and 5.07 points over zero‑shot CLIP for ViT‑B/16 and ResNet‑50, and maintains gains under ImageNet distribution shifts.

By Youngeun Seol, Jimin Shin, Heeseo Yoon, Uiwon Hwang
arXiv AI
Sep 25

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

The paper surveys 160 benchmarks from 2017‑2026 that evaluate predictive embodied intelligence, categorising them into policy suites, embodied agents, world‑model evaluation, and prediction‑to‑action bridges. It finds that most benchmarks are model‑agnostic, rarely compare Vision‑Language‑Action policies to world models, and seldom turn predictions into executed actions. The authors argue that the lack of benchmarks designed to directly test the closed‑loop advantage of world models prevents the field from answering whether such models truly improve robotic performance.

By Gaytri Jena, Kapil Wanaskar, Vinija Jain, Aman Chadha, Vasu Sharma, Amitava Das
arXiv AI
Sep 25

GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS

GHOST-Q evaluates how post‑training quantization affects visual grounding in vision‑language models. The study compares three 8B VLM families across FP16, INT8, and NF4 precisions, pairing predictions to measure how compression redistributes grounding successes and failures. While most quantized variants maintain overall accuracy, several exhibit significant changes in hallucination‑sensitive conditions, and memory savings do not always translate to lower latency.

By Saim Rehman, Muhammad Shafique