The paper introduces ActiveKD, a framework that combines active learning with knowledge distillation by using large vision‑language models as teachers. It leverages the structured prediction bias of these models to design Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in probability space rather than feature space. Experiments on 11 datasets show that ActiveKD improves all selection methods over a no‑distillation baseline, and PCoreSet consistently outperforms random sampling across multiple students and teachers.
arXiv:2505.07675v3 Announce Type: replace-cross Abstract: Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Rece...
arXiv:2606. 25488v1 Announce Type: new Abstract: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments.
arXiv:2605.10641v2 Announce Type: replace-cross Abstract: Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question...
The paper investigates data efficiency and selection in On‑Policy Distillation (OPD) for large language models. It shows that 1‑shot OPD—training on a single example—consistently improves performance, especially when the example is hard, and that longer chain‑of‑thought (CoT) paths drive the gains rather than token entropy. Based on these findings, the authors propose a simple hard‑example selection strategy that, using only eight carefully chosen hard examples, matches the performance of a 17,000‑example baseline across models from 1.5B to 7B parameters.
arXiv:2609.10311v1 Announce Type: cross Abstract: The lottery ticket hypothesis posits the existence of winning tickets: sparse subnetworks that, when trained in isolation from their original initial...
arXiv:2607. 05891v1 Announce Type: cross Abstract: Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training.
arXiv:2606. 18209v1 Announce Type: new Abstract: Dataset distillation (DD) has emerged as a prominent approach in data centric machine learning, aiming to synthesize compact training sets for efficient training by compressing the information in large datasets into a small number of synthetic samples.
arXiv:2606. 23897v1 Announce Type: cross Abstract: Prompt distillation compresses large vision-language models (VLMs) such as CLIP into lightweight student models by matching teacher predictions on unlabeled domain images.
The paper evaluates data‑efficient finetuning methods for vision foundation models, focusing on Visual Prompting (VP) and comparing it with Linear Probing, Full Fine‑tuning, and Low Rank Adaptation across different model families (Attention, Convolution, Mamba). It provides an empirical overview of how these models and transfer paradigms interact, broadening previous studies by including more architectures and tuning hyperparameters. The authors also investigate regularization techniques to enhance VP performance, aiming to offer practical guidelines for users.
arXiv:2609.10321v1 Announce Type: new Abstract: Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in curren...
PRISM is a training‑free framework that efficiently selects visual instruction data for multimodal large language models by addressing the anisotropy in visual feature distributions, which causes a Global Semantic Drift. By implicitly re‑centering visual semantics, PRISM removes the influence of global background features, cutting data‑selection and model‑tuning time to 30% of conventional pipelines while improving performance across eight multimodal and three language benchmarks, achieving a 101.7% relative gain over baseline models.
arXiv:2605. 15532v3 Announce Type: replace-cross Abstract: Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities, yet the prompts driving this process are typically chosen via simple heuristics or aggregated from off-the-shelf datasets.