The paper introduces a scalable multi-teacher distillation (MTD) framework for digital pathology, addressing the poor performance of standard MTD when many teachers are used. By learning teacher‑specific representations through techniques such as learnable teacher tokens, a tailored attention scheme, mixture‑of‑experts layers, and a contrastive loss, the method scales effectively, enabling compact student encoders that are up to 10× more efficient than larger teacher models while matching or surpassing their performance on 39 tile‑ and slide‑level tasks. The authors release a new collection of efficient foundation models, OMNI, trained from ten state‑of‑the‑art teachers and provide an analysis of the complementary teacher‑specific representations.
arXiv:2607. 14703v1 Announce Type: cross Abstract: Multiple instance learning (MIL) has become the main paradigm for whole-slide image (WSI) analysis in computational pathology.
arXiv:2610.02597v1 Announce Type: new Abstract: Vision foundation models such as DINOv2, SigLIP2, and MASt3R develop complementary capabilities from different pretraining objectives, yet their knowle...
arXiv:2607. 09526v1 Announce Type: cross Abstract: Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones.
arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.
arXiv:2607. 11257v1 Announce Type: cross Abstract: Pathology Foundation Models (PFMs) offer powerful Whole Slide Image (WSI) representations but suffer from massive computational costs.
arXiv:2606. 00928v1 Announce Type: cross Abstract: Multiplexed fluorescence microscopy improves tissue segmentation by providing complementary channels including nuclear (DAPI) and membrane (E-cadherin), that together encode richer spatial context than single-channel imaging alone.
The paper investigates how knowledge distillation (KD) applied at intermediate layers of a neural network can affect overfitting and model performance. While traditional KD focuses on the final output, this study explores block‑wise KD across eleven datasets, finding that on standard datasets the last block suffices, but on fine‑grained, data‑scarce settings intermediate supervision significantly improves accuracy. The authors also analyze optimal supervision granularity using attention maps, Centered Kernel Alignment, and Grad‑CAM, and examine teacher‑student fine‑tuning strategies.
arXiv:2609.37682v1 Announce Type: new Abstract: The rapid expansion of large-scale medical datasets and computational resources has driven significant progress in medical foundation models. Given the...
arXiv:2608.22066v1 Announce Type: cross Abstract: Attention-based multiple instance learning (ABMIL) using pathology foundation model embeddings is effective for slide-level tasks, but exhaustive inf...
The paper introduces IDeaL, a data‑free multi‑teacher distillation technique that generates teacher‑specific, improved samples using decorrelation losses at patch and image levels. By tailoring noise to each teacher, IDeaL produces strong student models that capture complementary teacher information and achieve results close to those distilled from real images. Experiments demonstrate that with only 1,000 images, students trained on IDeaL samples match or exceed the performance of students distilled from a 1,000‑image subset of ImageNet.
arXiv:2609.23561v1 Announce Type: new Abstract: Deploying efficient neural networks is essential in resource-constrained environments, yet compact models often sacrifice interpretability - a critical...
arXiv:2609.36407v1 Announce Type: new Abstract: Cross-resolution knowledge distillation aims to improve low-magnification whole- slide analysis by transferring high-magnification representations, yet...