The paper presents a method for cross‑architecture knowledge distillation from a fine‑tuned DINOv2 Vision Transformer teacher to a lightweight bidirectional Visual State Space Model (LVSSM) student for tea leaf disease classification. By addressing training‑stability issues with a progressive convolutional stem and gated selective‑scan block, the 4.45 M‑parameter student achieves a mean test accuracy of 95.41%—a 3.09‑point improvement over the teacher’s 92.32%—while using only one‑fifth of the teacher’s parameters. Ablation studies show that simple logit‑level distillation outperforms intermediate feature alignment, and the gains are specific to students that start below the teacher’s performance.
By Zibo Zhou, Zongsen Qiu, Rui Chen, Yujie Yao, Yue Zhou, Jianjun Wang
arXiv:2601. 22709v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.
By Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li
arXiv:2607. 18885v1 Announce Type: new Abstract: Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility, using a fixed trade-off weight tuned on curated ImageNet-1K.
By Micha{\l} Paw{\l}owicz
Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.
arXiv:2603. 13994v2 Announce Type: replace-cross Abstract: Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties.
By Hossein Adeli, Seoyoung Ahn, Andrew Luo, Mengmi Zhang, Nikolaus Kriegeskorte, Gregory Zelinsky
arXiv:2606. 05682v2 Announce Type: replace Abstract: Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments.
By Fangbo Tu, Junhua Zhao, Chi Liu, Xin Chen, Haifeng Wu, Jian Wan, Srinivasan Manoharan
arXiv:2606. 05682v1 Announce Type: cross Abstract: Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments.
By Fangbo Tu, Junhua Zhao, Chi Liu, Xin Chen, Haifeng Wu, Jian Wan, Srinivasan Manoharan
PhysMLLMs introduces physics-inspired spatial continuity priors into video multimodal large language models to address spatio‑temporal inconsistencies such as jitter, drift, and identity switches. The method, called Global Representation Prior Alignment (REPA‑Global), distills global visual representations from a frozen DINOv2 teacher during training, aligning student representations without affecting inference speed. Experiments on multiple video benchmarks show improved segmentation mask quality and cross‑frame consistency, especially for challenging scenarios involving small targets, fast motion, occlusion, and distractors, while maintaining comparable performance on single‑frame image segmentation and general VLM tasks.
By Siyao Yan, Bo Han, Jisheng Dang, Bimei Wang, Shude Wang, Hong Peng, Yulan Guo, Jianhuang Lai, Bin Hu, Tat-SengChua
Token-Budget Distillation (TBD) is a parameter‑efficient fine‑tuning framework that adapts video vision‑language models to a fixed token budget. It freezes the pretrained backbone, updates only LoRA adapters, and incorporates FlashVID visual token compression. TBD uses a dual‑path teacher‑student design with full‑token supervision and compressed student optimization, enabling the student to recover full‑token semantics while remaining efficient under aggressive token reduction.
By Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, Wenhao Wang
The paper introduces the task alignment proxy, a method that accelerates hyperparameter selection for merging models fine‑tuned on diverse vision tasks. It addresses the challenge of training heterogeneous decoders, which makes traditional downstream performance evaluation costly. By using the proxy, the authors demonstrate that model merging can be applied efficiently to multi‑task vision models beyond CLIP‑based classification.
By Pau de Jorge, C\'esar Roberto de Souza, Bj\"orn Michele, Mert B\"ulent Sar{\i}y{\i}ld{\i}z, Philippe Weinzaepfel, Florent Perronnin, Diane Larlus, Yannis Kalantidis
The paper presents a method for distilling a large 300‑million‑parameter geospatial foundation model (Prithvi‑EO‑2.0) into a compact 0.7‑million‑parameter EfficientViT‑B0 student for flood segmentation. By using the teacher to supervise additional unlabeled Sentinel‑2 imagery, the student’s training set expands without new manual labels, achieving competitive performance on Sen1Floods11 and STURM‑Flood while remaining smaller and faster. After quantization, the student runs as a 1.5‑MB INT8 TensorRT engine on a Jetson Xavier NX, processing 512×512 images in 5.57 ms with ~14 MB of memory.
By Fabian Schmalstieg, Karsten Mueller, Wojciech Samek
arXiv:2606. 04922v1 Announce Type: cross Abstract: Current prompt-based and adapter-based tuning of vision-language models (VLMs) is attractive for medical imaging, where clinical data sensitivity favors frozen backbones and annotations are limited.
By Tran Dinh Tien, Zhiqiang Shen