The paper introduces the Boot-and-Feedback (BooF) framework, which facilitates collaboration between multimodal large language models (MLLMs) and expert vision models for breast ultrasound (BUS) diagnosis. In the Boot Stage, the MLLM is guided by the BI-RADS lexicon and preliminary expert predictions to generate reliable textual descriptions, while the Feedback Stage fuses these descriptions with visual features using an Attention-Gated Cross-Modality Fusion Module, allowing the expert model to incorporate textual insights and filter out noise. Experiments on multiple BUS datasets show that BooF improves both diagnostic accuracy and interpretability compared to existing methods.
By Ming Cheng, Hongyu Sun, Zhaolin Chen, Jun Liu, Hossein Rahmani, Qiuhong Ke
arXiv:2406.05375v4 Announce Type: replace
Abstract: Root cause analysis (RCA) is crucial for enhancing the reliability and performance of complex systems. However, progress in this field has been hin...
By Lecheng Zheng, Zhengzhang Chen, Dongjie Wang, Chengyuan Deng, Reon Matsuoka, Haifeng Chen
PhysMLLMs introduces physics-inspired spatial continuity priors into video multimodal large language models to address spatio‑temporal inconsistencies such as jitter, drift, and identity switches. The method, called Global Representation Prior Alignment (REPA‑Global), distills global visual representations from a frozen DINOv2 teacher during training, aligning student representations without affecting inference speed. Experiments on multiple video benchmarks show improved segmentation mask quality and cross‑frame consistency, especially for challenging scenarios involving small targets, fast motion, occlusion, and distractors, while maintaining comparable performance on single‑frame image segmentation and general VLM tasks.
By Siyao Yan, Bo Han, Jisheng Dang, Bimei Wang, Shude Wang, Hong Peng, Yulan Guo, Jianhuang Lai, Bin Hu, Tat-SengChua
The paper introduces Split-then-Merge (StM), a new framework for generative video composition that improves control and tackles data scarcity. StM divides a large set of unlabeled videos into dynamic foreground and background layers, then self‑composes them to learn how subjects interact with varied scenes. The method employs a transformation‑aware training pipeline with multi‑layer fusion, augmentation, and an identity‑preservation loss, achieving superior performance over state‑of‑the‑art methods in both quantitative and qualitative evaluations.
By Ozgur Kara, Yujia Chen, Ming-Hsuan Yang, James M. Rehg, Wen-Sheng Chu, Du Tran
The paper introduces a dual‑judge evaluation protocol for vision‑language models in legally grounded tasks, pairing a 0‑10 quality judge with a strict binary semantic‑equivalence judge. Using a controlled UK traffic‑sign interpretation task, the authors analyze 4,680 evaluations across visibility and occlusion conditions, finding moderate association between judges and an asymmetric Type II error pattern that is most pronounced under heavy occlusion. The protocol requires only one additional LLM call and reveals quality‑trustworthiness signals that single‑judge methods miss.
By Su Myat Noe, Ha Thanh Nguyen, May Myo Zin, Ken Satoh
The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.
By Chen Ling, Tongwei Zhang, Hanqian Li, Nai Ding
arXiv:2607.19261v4 Announce Type: replace-cross
Abstract: Whole-slide image (WSI) diagnosis requires identifying diagnostically relevant regions, examining them across magnifications, and integrating...
By Dankai Liao, Tianyi Zhang, Yufeng Wu, Xinyue Zhang, Qiaochu Xue, Zeyu Liu, Dachun Zhao, Linghan Cai, Yueming Jin
The paper reviews the evolution of object‑counting techniques from class‑specific density regression to open‑vocabulary, foundation‑model‑backed counters that can handle visual and textual prompts. It highlights that current evaluation relies on a few saturated benchmarks, leading to models exploiting statistical regularities rather than true generalization. The authors propose a five‑axis taxonomy and audit the literature across domains such as microscopy, remote sensing, crowd counting, and agriculture, identifying six structural contradictions and outlining a roadmap for robust, multimodal evaluation protocols.
By Joana Konadu Owusu, Shivanand Venkanna Sheshappanavar
arXiv:2605.10426v3 Announce Type: replace-cross
Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing reasoning mecha...
By Minqing Huang, Yujiao Xiang, Zihan Liang, Jiajie Huang, Jingqi Wang, Yuheng Zhou, Zhi Xu, Feiyang Tan, Hangning Zhou, Mu Yang, Gong Che
arXiv:2608.24430v1 Announce Type: new
Abstract: Responsible deployment of face verification systems requires more than accurate decisions: systems should also provide interpretable and auditable evid...
By Ana Estrada-Real, Lydia Alapatt, Christoph Busch, Christian Rathgeb
HMGCLIP is a unified multimodal embedding framework that uses a heterogeneous hypergraph to capture both fine‑grained and coarse‑grained product attributes. By mining structure‑aware hard negatives and aligning multi‑granular semantics at relation and hyperedge levels, it enables a dual‑granularity inference mechanism that dynamically fuses attribute evidence. Experiments on a new fine‑grained e‑commerce dataset and the public MAVE benchmark show that HMGCLIP outperforms strong multimodal encoders, MLLMs, and e‑commerce baselines.
By Qiuyu Zhu, Yi Gao, Zhichao Wan, Mingyang Ma
SonarLLM is a multimodal large language model that treats sonar as a native perceptual modality, combining a sonar‑specific encoder, physics‑aware feature enhancement, and reliability‑aware hierarchical fusion to align acoustic structure with optical semantics. The authors introduce SonarBench, a benchmark covering recognition, counting, visual question answering, and captioning across sonar‑only, optical‑only, and fusion settings, enabling controlled measurement of cross‑modal complementarity. SonarLLM achieves 72.0% macro accuracy on sonar‑only tasks and 68.7% under fusion, outperforming baselines by significant margins and demonstrating increasing fusion gains as optical visibility degrades.
By Cong Su, longxuan ma, Ling Dong, Guofeng Tang, Weijie Yin, Haohui Chen, Zhengtao Yu
The paper introduces D3-Omni, a balanced and decoupled benchmark designed to diagnose fine‑grained multimodal understanding in OmniJudges that evaluate text‑to‑image, text‑to‑video, and text‑to‑speech generation. D3-Omni covers 53 orthogonal binary dimensions across 10,671 samples, using fixed positive seeds and controlled prompt rewriting to generate negatives, thereby ensuring each error can be attributed to a single capability. The benchmark’s dual‑balanced, decoupled, and dynamic design achieves near 1:1 per‑dimension parity and a uniform total‑score distribution, revealing that strong OmniJudges often miss modality‑related failures and treat distinct attributes as a single decision, masking systematic blind spots.
By Guangzheng Hu, Ziyue Jiang, Weixu Qiao, Lixin Zhang, Jianye Kang, Yuru Wu, Rong Bao, Niantong Li, Wei Wang, Ziyi Cheng, Xinfa Zhu, HangRui Hu, Ting He, Bing Zhao, Lin Qu, Hu Wei, Jin Xu
The paper introduces NAIMA, a guided depth super‑resolution framework that leverages global contextual semantic priors from pretrained vision transformer token embeddings. Its Guided Token Attention (GTA) module uses depth encodings as queries to attend over semantic tokens, with a zero‑initialized gate controlling the influence of semantic evidence. NAIMA achieves competitive in‑distribution performance while delivering superior cross‑dataset generalization without relying on decoded priors or auxiliary objectives.
By Tayyab Nasir, Daochang Liu, Ajmal Mian
The paper investigates the blending ratio used in few‑shot adaptation of vision‑language models, which combines a zero‑shot text prototype with the mean of labeled image features. It shows that the theoretically optimal ratio—derived from a closed‑form mean‑squared error minimizer—does not align with the ratio that actually maximizes performance, falling short by an average of 8.5 points. Moreover, a leave‑one‑out estimate on the support set achieves near‑oracle performance, and validation‑free linear probes outperform even oracle‑tuned blends, indicating that the hyperparameter can be set near‑optimally without external validation data.
By Liangzhi Li, Bowen Wang, Yiming Qian, Thorsten Neumann, Xia Xie, Guangshun Li
arXiv:2608.24795v1 Announce Type: new
Abstract: Recently, the rapid advancement of multimodal domains has driven a data-centric paradigm shift in graph ML, transitioning from text-attributed to multi...
By Xunkai Li, Zekai Chen, Zhengyu Wu, Henan Sun, Daohan Su, Guang Zeng, Hongchao Qin, Rong-Hua Li, Guoren Wang
The paper tackles the challenge of predicting student engagement from online tutoring videos, noting that engagement is a complex, multidimensional construct influenced by behavioral, emotional, and cognitive states. By analyzing the CASED dataset, the authors highlight the difficulty posed by high inter‑person variability and subjective annotations. They propose a multimodal framework that fuses implicit spatiotemporal features from pretrained video, audio, and image encoders with structured behavioral cues such as head pose, gaze, facial action units, emotion, and wavelet‑based audio features, integrating them via a Perceiver IO bottleneck and modeling participant personalities with variational posteriors. The system employs evidential regression and spectral‑normalized Gaussian process classification heads to provide uncertainty‑aware predictions, achieving competitive performance on the CASED challenge test set while offering well‑calibrated uncertainty metrics.
By Alperen Kantarci, Visvanathan Ramesh, Gemma Roig
The paper introduces a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), examining how varying amounts of initial target information and dialogue affect performance. Experiments across four visual contexts and interaction protocols show that current LVLMs lag behind human baselines, especially when no initial description is given and information must be gathered through questions. The study also finds that LVLMs are poorly calibrated, often overestimating confidence, and that interactive grounding remains a significant challenge requiring visual matching, information seeking, and synthesis.
By Zhengxiang Wang, Owen Rambow
The paper introduces a new evaluation protocol for deep multivariate imputation models on wearable device data, addressing the issue of structured missingness where sensor features drop out together. Using a Garmin smartwatch dataset from an epilepsy patient, the authors generate realistic block-missing patterns from training data and show that matching the training protocol to this distribution reduces BRITS’ mean absolute error by 43%. They also extend BRITS with time‑of‑day encoding and compare it to linear interpolation and SAITS, finding that no single model dominates and that model rankings vary with evaluation design.
By Skye Goodman, Roussel Desmond Nzoyem, Leandro Junges, Peter Kissack, Yasser Qureshi, Amberly Brigden, Jeff Clark, Nawid Keshtmand
arXiv:2608.24535v1 Announce Type: new
Abstract: Data visualizations are widely used for communicating information, but they are also vulnerable to intentional manipulations that induce misleading int...
By Xiaotian Zhang, Huayuan Ye, Haiyang Zhang, Chenhui Li, Changbo Wang, Sicheng Song