EgoMemReason is a new benchmark for week‑long egocentric video understanding that focuses on memory‑driven reasoning rather than simple perception tasks. It tests three memory types—entity, event, and behavior—across 500 questions, each requiring evidence from an average of 5.1 video segments and 25.9 hours of backtracking. Evaluation of 17 models shows that even the best achieves only 39.6% accuracy, highlighting the difficulty of long‑horizon memory in multimodal systems.
By Ziyang Wang, Yue Zhang, Shoubin Yu, Ce Zhang, Zengqi Zhao, Jaehong Yoon, Hyunji Lee, Gedas Bertasius, Mohit Bansal
FedCoRe is a federated learning framework that addresses missing modalities in healthcare by learning representation- or logit-space corrections instead of generating synthetic data. In a MIMIC-derived respiratory deterioration task, the method uses paired examples where a modality is present or absent to train a completion module, achieving partial recovery of performance lost when modalities like ECG or CXR are hidden. The approach emphasizes validation-gated deployment, ensuring that completion is only applied when paired examples and validation evidence support the presence of the missing modality.
By Holger R. Roth, Ziyue Xu, Peter Cnudde
The paper introduces the Graded Color Attribution (GCA) dataset, a benchmark that tests whether Vision‑Language Models (VLMs) and humans can articulate and follow a threshold rule for labeling objects by color. In experiments, humans consistently adhere to their stated rules, while VLMs—despite accurately estimating color coverage—often violate their own introspective rules, especially when world‑knowledge priors are present. This discrepancy highlights a miscalibration in VLM self‑knowledge that differs from human cognition.
By Jonathan Nemitz, Carsten Eickhoff, Junyi Jessy Li, Kyle Mahowald, Michal Golovanevsky, William Rudman
Event-Causal RAG (EC‑RAG) is a lightweight retrieval‑augmented framework designed for reasoning over ultra‑long and streaming videos. It segments video streams into semantically complete events using a dual visual‑audio sentinel mechanism, representing each event as a State‑Event‑State (SES) structure that captures pre‑event, event, and post‑event states. During question answering, bidirectional graph retrieval accesses relevant predecessor and successor events from a dual vector‑graph memory, and answers are generated using both this structured memory and the corresponding video evidence. The authors also introduce ECV‑1H, an hour‑scale long‑video QA benchmark with over 150 hours of untrimmed video and 1,251 human‑annotated QA pairs, where EC‑RAG achieves significant accuracy gains across multiple video foundation models while maintaining efficient streaming memory usage on a single RTX 5090 GPU.
By Peizheng Yan, Yu Zhao, Liang Xie, Juntong Qi, Mingming Wang, Erwei Yin
The paper introduces TTSD‑FAR, a test‑time self‑distillation framework that adapts large video‑language models to missing‑modality scenarios in emotion recognition. A frozen teacher trained on complete modalities guides a low‑rank student, while Fisher‑Anchored Restoration monitors Fisher information to prevent drift and restore the student when distribution shifts occur. Experiments on MELD, DFEW, and BAH with up to 50% missing modalities show TTSD‑FAR consistently outperforms entropy‑based adaptation, retrieval‑augmented generation, and perplexity‑based generation, maintaining performance over long adaptation horizons.
By Muhammad Haseeb Aslam, Alessandro Koerich, Marco Pedersoli, Ali Etemad, Eric Granger
arXiv:2608. 18339v1 Announce Type: cross Abstract: Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.
By Qi Yu, Zhichen Zeng, Katherine Tieu, Xiyuan Yang, Ruizhong Qiu, Yuchen Yan, Lihui Liu, Yanjun Zhao, Lingjie Chen, Jingrui He, Hanghang Tong
The paper investigates whether large, instruction‑following Vision‑Language Models (VLMs) can reliably perform zero‑shot image privacy classification. It compares three open‑source VLMs to specialized privacy models on two public benchmarks, evaluating accuracy, robustness to image degradations (compression, lighting changes, noise), inference speed, and parameter count. The findings show that while VLMs remain robust to perturbations, they are less accurate and significantly slower than smaller, purpose‑built privacy models, indicating that scaling alone does not guarantee effective privacy classification.
By Alina Elena Baia, Alessio Xompero, Andrea Cavallaro
DentAgent is an evidence‑centric multi‑agent framework designed for multimodal dental reasoning. It coordinates five specialized agents that process domain knowledge, radiographs, intraoral photographs, and 3D dental data, converting observations into structured evidence records. The Evidence Blackboard tracks coverage, gaps, and conflicts before generating responses, and the system outperforms senior specialists by 17.3 percentage points on multi‑label diagnosis across four benchmarks.
By Zijie Meng, Xiwei Dai, Yixuan Tang, Jin Hao, Yang Feng, Fudong Zhu, Xiaoqiang Liu, Shaosheng Cao, Zuozhu Liu
The paper introduces Role-Conditioned Sub-Token Routing (RoleSub), a method that compresses the value representations of retained tokens in Vision‑Language‑Action models. By partitioning each token’s value into orthogonal groups and routing them based on token features, a latent role, and language context, RoleSub can also compress language values. Experiments on OpenVLA‑OFT‑7B show that, at matched visual‑KV budgets, RoleSub outperforms token‑only control in most settings and reduces total KV to 9.2–11.3% of the original while maintaining strong control performance.
By Wei Jiang, Wei Wang
The review examines how Large Language Models (LLMs) enhance nanophotonics design by providing semantic interfaces, code generation, and tool orchestration. It traces the evolution from classical neural networks to transformer-based models and categorizes LLM applications into surrogate models that map structure to spectrum and agentic systems that generate code and orchestrate simulations for closed-loop optimization. The article also highlights potential cross-disciplinary uses of LLMs in materials science and wireless communications, and envisions future multimodal foundation models that actively collaborate in autonomous scientific discovery.
By Huanshu Zhang, Kegeng Tang, Lei Kang, Sawyer D. Campbell, Zihao Wang, Douglas H. Werner
The paper investigates how Vision Language Models (VLMs) can be fooled by small, human‑imperceptible changes to images. It introduces a gradient‑based attack that targets only the vision encoder, reducing computational cost while still effectively disrupting both untargeted and targeted multimodal alignment. Experiments on open‑source VLMs such as Qwen2.5‑VL, Granite‑Vision, FastVLM, and Phi‑3.5‑Vision demonstrate that these perturbations can significantly alter the models’ textual outputs.
By Ilan Zini, Boussad Addad, Katarzyna Kapusta
GS‑VLA introduces a lightweight, plug‑and‑play framework that uses a 4 M‑parameter 3D‑Gaussian canonicalizer to adapt frozen Vision‑Language‑Action (VLA) policies to viewpoint shifts without retraining the policy. By treating viewpoint changes as a localized novel‑view synthesis problem under a locality assumption, the method normalizes observations through a scene‑ and policy‑independent disocclusion task. Experiments on the LIBERO benchmark demonstrate that GS‑VLA recovers a large portion of performance lost due to camera displacement, improving results across different policy architectures, unseen task suites, and perturbation scales.
whyItMatters":"The approach offers a computationally efficient alternative to costly fine‑tuning or generative augmentation, enabling robust VLA deployment in real‑world settings where camera configurations may vary."
By Yechan Park, HyunJin Kim
The paper studies how prompt quality affects vision‑language models in the agrifood domain. It evaluates Zero‑Shot Prompt Ensembling (ZPE) on CLIP and SigLIP across four datasets, showing that ZPE offers limited gains on in‑distribution data but significantly improves accuracy and calibration when the domain shifts. The authors also introduce Prompt‑based Inconsistency Detection (PID), which uses prompt disagreement to detect failures under severe domain shift, outperforming standard confidence measures.
By Andrea Morales-Garz\'on, Salvador L\'opez-Joya, Miguel L\'opez-P\'erez, Maria J. Martin-Bautista
The paper evaluates open-source OCR, LLM, and VLM systems on a high‑risk public sector task: extracting structured data from student application documents. Results show that VLMs generally outperform OCR+LLM pipelines, yet only 4 of 35 configurations achieve F1 scores above 0.5, with most combinations scoring below 0.25. Model size and input quality, especially preserving OCR structure, are critical factors influencing performance.
By Elias Schubert, Felix Bie{\ss}mann
The paper reviews how large language models are applied in mental health, covering areas such as social media analysis, clinical conversational agents, therapy support tools, prompt engineering, and multimodal learning. It synthesizes interdisciplinary studies that use social media posts, electronic medical records, and multimodal inputs to detect depression, assess suicide risk, provide personalized therapy, and generate psychoeducational content. The review also discusses advances in model interpretability, annotation strategies, multimodal fusion techniques, and highlights ethical, sociotechnical, and regulatory challenges while proposing frameworks for safe, equitable, and accountable deployment.
By Yisong Chen, Yifan Gao, Sijing Yu, Chuqing Zhao, Yang Lu
The paper examines how multilingual medical adaptation affects the internal representations of Whisper ASR models. By comparing various fine‑tuning strategies—zero‑shot decoding, English‑only, German‑only, two‑stage EN→EN+DE, and direct EN+DE fine‑tuning—it shows that fine‑tuning significantly improves performance, with the best model varying by setting. Layer‑wise encoder analysis reveals that English medical fine‑tuning drives the main representation shift, while multilingual continuation largely preserves the adapted space, and that domain and language signals remain recoverable across layers.
By Souranil Kahali, Rituparna Bose, Abner Hernandez, Tomas Arias-Vergara, Andreas Maier, Ning Ma, Paula Andrea Perez-Toro
The paper introduces an adaptive jailbreak attack framework that evaluates both cascaded pipelines and end‑to‑end large audio‑language models (LALMs) under a unified setting. It employs a feedback‑guided mutation engine to automatically generate and refine jailbreak candidates across textual prompts and audio perturbations, thereby broadening attack diversity. Experiments on six audio‑based systems show that both paradigms remain highly vulnerable, with the framework achieving higher attack success rates than existing methods.
By Linghan Huang, Bo Li, Huaming Chen, Kim-Kwang Raymond Choo
The paper introduces HN-CLIP, a new objective for dense-caption retrieval that adapts similarity margins per negative example using the text encoder’s own geometry. By adding a detached caption‑similarity matrix to the negative logits, HN‑CLIP addresses the issue of near‑duplicate captions that cause premature loss saturation in InfoNCE training. Experiments on four benchmarks show that HN‑CLIP outperforms leading methods by 2.4–4.3 R@1, trains 2.4× faster than GOAL and 5.4× faster than StructXLIP, and achieves the best full‑data baseline with only 20% of the training data.
By Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang
The paper introduces MedUAG, a unified medical multimodal model that supports both understanding and generation tasks. It presents MedUAGCorpus, the largest dataset of over 6 million instances across 14 imaging modalities, and MedUAGBench, a benchmark covering 12 diverse generation tasks with standardized protocols. Experiments show that MedUAG performs strongly across many medical understanding and generation tasks, setting a competitive baseline for future medical multimodal systems.
By Zijie Meng, Yuncheng Zhang, Hualiang Wang, Yitian Tang, Xiaotang Gai, Chen Shen, Songtao Jiang, Shaosheng Cao, Jian Wu, Xian Wu, Zuozhu Liu
The paper discusses how recent progress in deep learning has improved Natural Language Processing and Vision‑Language Models, but also introduced new vulnerabilities, particularly backdoor attacks that threaten security. It explores methods for analyzing, detecting, and designing such attacks in both NLP and VLMs. Additionally, it proposes efficient multimodal representation techniques aimed at clinical and medical imaging applications.
By Weimin Lyu