arXiv AI

Model Annealing: Data-Efficient Learning through Review and Consolidation

Model Annealing is a data‑efficient learning method that, after an initial memorization phase, spends the remaining training budget on review and consolidation. The training is split into episodes with cyclic learning rates; each episode learns from true labels, tempered predictions stored in a memory of past models, and adversarial ‘dreams’ where the student and memory disagree most. Across several datasets and architectures, Model Annealing improves top‑1 accuracy, likelihood, calibration, and robustness compared to standard cross‑entropy training and matches snapshot distillation while adding no inference overhead.

arXiv AI
Oct 1

Unmerge: Efficient Machine Unlearning via Task Arithmetic

The paper introduces Unmerge, an efficient machine unlearning algorithm that treats unlearning as the inverse of task arithmetic. By representing the forget component as a low‑rank basis at each layer, Unmerge optimizes three goals—matching the merged vector, suppressing leakage, and bounding correction size—to limit forget leakage and retain damage. Experiments on ResNet‑50, ViT‑S/16, and Llama‑3.2‑3B show significant performance gains over existing methods while maintaining privacy and feature‑distribution fidelity.

By Haoran Tang, Andrew Tan, Rajiv Khanna
arXiv AI
Oct 12

Useful to Whom? The Easy-versus-Coverage Boundary Moves with the Learner

The study investigates whether the usefulness of training samples depends on the data itself or on the learner. By fixing selected subsets and varying the learner, the authors show that the crossover point—where easy samples give way to coverage samples—shifts with changes in model width, input resolution, and architecture. This indicates that the boundary between easy and coverage samples is learner‑dependent rather than data‑dependent.

By Yangze Liu, Xiao-Long Yin, Zhongyi Han
arXiv Computer Vision
Aug 24

When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

arXiv:2608.21098v1 Announce Type: new Abstract: Fusing prior knowledge with data-driven learning is attractive where data is scarce, yet no controlled account says when it helps, is redundant, or har...

By Ahmad AlMughrabi, Albert Clop, Benjamin Busam, Ricardo Marques, Petia Radeva
arXiv Machine Learning
Sep 30

Scaling Zero-Order Pretraining through Model Sharding

arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...

By Francois Chaubard, Mykel J. Kochenderfer, Chris R\'e
arXiv Computer Vision
Nov 2

Scout-Refine-Correct: Teaching Flow Models to Refine Their Own Samples

Scout-Refine-Correct (SRC) is a training recipe that equips flow-matching image generation models with an inference-time refinement capability. During sampling, the model performs a scout rollout to predict a full sample, uses this prediction as a self-conditioning signal, and then takes the actual step; this process can be repeated for multiple refinement rounds. On ImageNet-256, SRC improves LightningDiT-B’s FID from ~14.5 to ~7.26 with 550 normalised flow evaluations, matching a best-of‑N baseline that relies on an oracle verifier, and further reduces FID to 3.81 when combined with best-of‑N.

By Sergey Karpukhin, Aibek Alanov
arXiv AI
Jul 22

Soft-TransFormers for Continual Learning

arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.

By Haeyong Kang, Chang D. Yoo
arXiv AI
Oct 2

Useful to Whom? Sample Value Is Defined Only Relative to the Learner

The paper investigates how the usefulness of training samples, as determined by coreset selection, depends on the learner rather than just the data. Experiments on ImageNet-100 and ImageNet-1k show that changing model width, input grid, stride, and architecture (e.g., ResNet vs. ViT) shifts the crossover point where different selection criteria (easy-first vs. geometric coverage) become optimal. These findings demonstrate that the relative value of a fixed subset of samples varies with the target learner’s capacity and structure, and that selection strategies must be tuned to the specific model they will train.

By Yangze Liu, Xiao-Long Yin, Zhongyi Han