ShallowStream is a framework for streaming video understanding that uses the shallow layers of a multimodal large language model (MLLM) to encode frames and build a lightweight index. During streaming, it maintains an always‑on index via the KV cache of shallow layers, and at query time it scores context frames using shallow‑layer attention and selects diverse evidence for answering. The approach matches the performance of leading streaming methods while cutting per‑frame prefill latency and 10‑second end‑to‑end latency by up to 52.1× and 11.9×, respectively.
By Jitai Hao, Ke Yang, Qiang Huang, Jun Yu
DocHop is a new benchmark that tests multimodal large language models on integrated chart‑context reasoning within document‑style images. The benchmark presents narrative text that imposes multi‑step compositional constraints, while charts supply the data needed to answer questions grounded in semantic reference labels. It contains 2,074 examples across six task categories, generated via a stochastic logic‑first pipeline that controls reasoning depth and visual density, and shows a large performance gap between humans (over 90% accuracy) and the best models (62.83%).
By Zhuoran Yu, Le Thien Phuc Nguyen, Jaden Park, Xinyi Gu, Zexue He, Soochahn Lee, Rogerio Feris, Yong Jae Lee
The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.
By Kabir Thayani
FPCO-Dialog is a benchmark designed to evaluate how vision‑language models correct and cooperate when faced with repeated false premises in multi‑turn dialogues. It includes 1,080 images and 10,800 question turns, organized by visual complexity, object category, and false‑premise class, and follows a 10‑turn protocol that introduces a correct dialogue prefix followed by repeated false‑premise expressions. The benchmark uses a model‑agnostic protocol and a correction‑rate metric (CorrTP@K) scored by two independent detectors, revealing significant differences in correction behavior across 20 commercial and open‑source VLMs.
The paper introduces Coupled Tensor‑Tensor Completion (CTTC), a new framework that incorporates side information in tensor form to enhance tensor completion tasks. CTTC leverages hidden connections among multimodal tensors and is grounded in distance metric learning and group theory. Experiments on the DTD and LINCS datasets show that CTTC outperforms existing methods such as HaLRTC, CTRC, Cell, and NTDDR in both run‑time and root‑sum‑of‑errors accuracy for predicting drug effects.
SafeEvolve is an experience-driven framework that co‑evolves a harness and policy to improve safety alignment for LLM‑based agents. It uses on‑policy trajectory safety evidence to update safety prompts and hierarchical skills, producing auditable harness artifacts. The policy is trained via a two‑stage SFT‑RL pipeline that bootstraps with the evolved harness and then refines behavior through verifier‑decomposed rewards, yielding a better safety‑utility tradeoff on benchmarks such as AgentDojo.
SPADE is a labelled, multi‑modal, simulation‑based dataset for detecting attacks on Signal Phase and Timing (SPaT) messages from the perspective of connected vehicles. It contains 1.89 million timestep records generated by injecting six classes of application‑layer attacks and one benign class into the SAE J2735 SPaT protocol, across multiple intersection geometries, operating conditions, and random seeds. Each record fuses SPaT fields, onboard camera confidence scores, and cooperative V2V peer data into 40 features, enabling deep‑learning intrusion detection systems to distinguish deliberate attacks from environmental noise.
UTP‑Bench is a new benchmark that evaluates large language models on uncertainty‑aware travel planning, incorporating real‑world data from 504 Indian cities and empirical delay and crowd patterns. It introduces three metrics—Buffer Adequacy Score, Crowd‑Aware Timing Score, and Transport Delay Absorption Score—to measure how well generated itineraries remain robust under stochastic conditions. Experiments show that current LLMs lag behind human planners in buffering, delay‑aware scheduling, and crowd sensitivity.
The paper introduces a multimodal framework that learns subcellularly resolved cell embeddings by integrating RNA expression profiles, protein sequence representations, and protein structural information. It uses a cross‑attention architecture to model interactions across distinct subcellular compartments, producing embeddings that capture both molecular expression patterns and functional protein properties. This approach is presented as the first to jointly incorporate transcriptomic data, protein sequences, and structural knowledge within a unified cross‑modal learning paradigm.
Counter‑GEO‑Bench is a new defense benchmark that tests how well systems can resist misinformation generated by generative engine optimization (GEO). It contains 247 human‑verified queries paired with both information‑preserving and information‑distorting GEO rewrites, and evaluates defenses on attack success rate, false positives, and answer quality across three large language models. Existing off‑the‑shelf defenses reduce attack success by at most 5.7%, while a lightweight baseline, C‑GEO Guard, cuts it by 47.6% with minimal loss of utility.
LeakageBench is a new benchmark consisting of 500 document images with 11,954 GDPR‑aligned PII annotations, designed to evaluate document‑level redaction risk. Unlike existing text‑centric PII benchmarks, it measures whether a page remains unsafe if any identifier is missed, using entity‑level F1, group‑wise leakage, and document‑level leakage metrics. Experiments show that even advanced OCR pipelines and vision‑language models improve localization but still leave a high proportion of pages unsafe for release.
TAME introduces a Temporal-Aware Mixture-of-Experts framework for Text-Video Retrieval that enhances CLIP-based models by incorporating frame-level structure and temporal relations. It adds sparse Mixture-of-Experts layers with frame-consistent routing, Frame-Temporal tokens for global cross-frame aggregation, and a Cross-Temporal Interaction and Aggregation module to refine sentence-video similarities. Experiments on multiple TVR benchmarks show consistent performance gains, such as a 4.0 R@1 improvement on MSR‑VTT over CLIP4Clip.
The paper demonstrates a lightweight method to adapt general-purpose vision‑language models (VLMs) for multispectral and synthetic aperture radar (SAR) image understanding. By rendering each observation as five optical views and one SAR view, naming them in the prompt, and applying LoRA to the language network and selected visual transformer blocks, the authors enable VLMs to process band composites, spectral indices, and radar backscatter without retraining a new foundation model. On a balanced six‑class land‑cover benchmark from BigEarthNet‑v2, the adapted Qwen3‑VL achieves a micro F1 score of 0.8275, and the same protocol improves four other VLMs and transfers to flood verification and captioning tasks.
TC-Next is a multimodal deep learning model that forecasts tropical cyclone track and intensity 6–24 hours ahead by combining a foundation model’s atmospheric forecast fields with GridSat infrared satellite imagery. Trained solely on GraphCast forecasts for the Western Pacific, it reduces track error by 15–44 % and intensity error by a factor of 3–6 compared to the rule‑based tracker TempestExtremes, and maintains superior performance when applied zero‑shot to other forecast systems such as Pangu‑Weather, IFS HRES, and WeatherNext Cyclones. Ablation studies confirm that incorporating the additional satellite modality consistently improves tracking accuracy at all lead times and enhances intensity predictions, especially at longer horizons.
EmbodiedSkills is a unified framework that treats each skill decision as an execution proposal, checking prerequisites and verifying outcomes during long‑horizon vision‑language‑action tasks. It connects high‑level skill selection, bounded low‑level VLA execution, and post‑action verification through a fixed executable‑skill interface, enabling easy replacement of low‑level policies and recording of structured trajectories for supervision and adaptation. Instantiated with Qwen3‑VL and OpenPI/pi0.5 on RoboTwin 2.0 and LIBERO, the framework achieves high success rates (86.20% and 97.40% respectively) and demonstrates effective memory‑dependent task performance.
By Wei Wang, Wenqiao Zhang, Yutong Lin, Yuqian Yuan, Tianwei Lin, Jinhao Mao, Zhenxuan Fan, Mingjian Gao, Yang Dai, Wentong Li, Zheqi Lv, Zheng Dong, Yingjie Niu, Jiaqi Zhu, Jun Xiao, Chao Li, Yueting Zhuang
REFACTOR‑VLA is a wake/sleep framework that learns reusable motor‑program abstractions for vision‑language‑action models. In its sleep phase, it clusters motor‑program fragments using a Behavioral‑Equivalence Kernel derived from rollouts of a latent world model, while its wake phase emits typed lambda terms that a library‑conditioned action decoder consumes. Experiments on LIBERO show that adding an auxiliary supervised contrastive loss during world‑model warm‑up significantly improves clustering quality, outperforming the strongest published baseline across all four task suites.
By Riyaaz Shaik, Chandru Venkataraman
arXiv:2609.00746v1 Announce Type: new
Abstract: Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that re...
By Minsik Choi, Geewook Kim, Young Geun Kim
arXiv:2609.01487v1 Announce Type: cross
Abstract: Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable chan...
By Xiaofang Yang, Ziqi Miao, Dianbo Sui, Jing Shao, Lijun Li
The paper introduces VGEBench, a new benchmark for evaluating Vision‑Language Models (VLMs) on generalizable, visually grounded exploration of household devices. Unlike existing datasets that rely on static images or annotated trajectories, VGEBench employs a logic‑driven state machine to simulate multi‑turn interaction loops, requiring agents to actively perceive, act, and refine their actions to achieve goals. Experiments show that current VLMs struggle to translate semantic knowledge into physical execution and to maintain long‑horizon state tracking.
By Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che, Heyan Huang, Yuhang Guo
arXiv:2609.00068v1 Announce Type: cross
Abstract: Medical AI is moving beyond recognition towards clinical dialogue and longitudinal prediction. Yet a central question remains: how would a patient's...
By Shuo Wang, Yike Guo