The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Compact instruction-following rerankers are attractive for deployment, but conventional distillation pipelines typically train students by offline imitation of teacher outputs on a fixed set of exampl...
The paper introduces a two‑stage training framework for compact instruction‑following rerankers. Stage 1 strengthens a 4B teacher reranker with off‑policy GRPO using LLM‑judge feedback on 88K examples, while Stage 2 trains a 1B student by sampling its own rankings and receiving soft teacher‑derived rewards, blending exploration with knowledge transfer. The method achieves superior nDCG and MRR scores on MAIR‑11 and MAIR‑Full benchmarks, outperforming offline distillation baselines and larger RL‑trained rerankers.
SCOPE-OPSD introduces a Fisher‑conditioned privileged subspace for on‑policy self‑distillation (OPSD) that projects the teacher‑student residual onto a frozen rank‑64 factor derived from residual covariance and language‑model‑head Fisher sensitivity. The method adds no extra rollouts or inference modules and, across multiple Qwen3 model checkpoints and trajectory lengths, consistently matches or surpasses pure OPSD and a matched random baseline, achieving significant gains in most settings. A cross‑fitted diagnostic shows a 4.40‑fold increase in captured privileged‑gap compared to the random orientation.
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations. "whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
GEAR is a two‑stage framework that distills tabular foundation models into lightweight MLP or tree‑based predictors for efficient CPU deployment. In the first stage, synthetic covariates are used as teacher‑query locations to train the student on soft TFM targets, expanding coverage beyond observed rows. The second stage re‑anchors the student to the target distribution using real labels and out‑of‑fold teacher predictions, preventing self‑labeling leakage and improving performance. Experiments on TALENT and TabArena show that GEAR‑distilled MLPs outperform supervised MLPs by up to 2.00 AUC points on binary tasks and 1.35 on multiclass tasks, and also outperform CatBoost, while dramatically reducing inference time and memory usage.
The paper investigates when external priors (teacher models, LLMs, or domain experts) help low‑resource structured prediction, introducing Ranking Prior Alignment (RPA) as a model‑agnostic objective that aligns a model’s ranking distribution to an external prior via a temperature‑scaled KL loss. Experiments on tree‑ensemble and multiple‑instance‑learning tasks show that a prior’s benefit depends on its quality (rank correlation with labels) and the granularity of alignment: high‑quality priors improve performance in cross‑sample tree alignment but can hurt within‑bag MIL alignment, while zero‑information priors act as entropy regularization. The authors propose a two‑factor decision rule and a protocol to separate information transfer from regularization.