arXiv Computer Vision
GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Computer Vision
Sep 25
IronViT: Toward Efficient Generalist Visual Representation Learning
IronViT proposes a new approach to building efficient generalist vision encoders by first consolidating the knowledge of multiple specialist teachers into a softmax attention bridge and then transferring this consolidated representation to a hybrid softmax‑linear attention architecture. This two‑stage distillation process, supported by a curated data pipeline, allows the model to capture semantic, spatial, language‑aligned, and action‑relevant cues while avoiding the high‑resolution cost of traditional softmax attention. Across tasks such as recognition, retrieval, dense prediction, multimodal understanding, and robotic learning, IronViT matches or exceeds the performance of leading specialist and generalist encoders, with the hybrid encoder offering increasing efficiency at higher resolutions.
arXiv AI
Jul 13
ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts
arXiv:2607. 09526v1 Announce Type: cross Abstract: Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones.
arXiv AI
Jun 19
Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts
arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.
arXiv Computer Vision
Aug 26
DDMS: Discriminative Distillation of Multi-view Foundational Features into Single-view Models
arXiv:2608.23850v1 Announce Type: new Abstract: Foundational visual features such as DINO have played a critical role across modern computer vision, and have recently become key components in multi-v...
arXiv Computer Vision
4d ago
Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring
Dyna‑DINO introduces a curriculum for Vision Transformer (ViT) knowledge distillation that uses the teacher’s intermediate feature maps as progressively harder targets, enabling a student to build foundational representations before tackling higher‑level abstractions. The approach accelerates convergence and improves performance across multiple tasks: on ImageNet‑100 the distilled ViT‑S reaches 90.1% accuracy (+12.24% over baseline), while on ImageNet‑1K it yields +3.9% and +6.09% gains on Oxford and Paris retrieval, +1.93% on semantic segmentation, and notable classification improvements. Additionally, the curriculum reduces training FLOPs by 25.1% and training time by 21% on ImageNet‑100 through early‑stopping of teacher inference.
arXiv AI
Jul 17
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
arXiv:2607. 14703v1 Announce Type: cross Abstract: Multiple instance learning (MIL) has become the main paradigm for whole-slide image (WSI) analysis in computational pathology.