← Back to all news
arXiv AI By Alejandro Rodriguez Dominguez, Muhammad Shahzad, Xia Hong

Teacher-Anchored Selection of Post-Training Quantized Models under Domain Shift

Read the original on arXiv AI →

The paper investigates how to choose the best quantized model from a family of compressed versions when target labels are scarce or unavailable. It finds that a simple rule based on minimum teacher distortion consistently selects the same eight‑bit, per‑channel, unclipped configuration, though this does not minimize empirical target cross‑entropy. The study also shows that confidence‑based estimators perform poorly in overconfident regimes, while output‑distribution estimators can outperform the teacher in some architectures, and that combining distortion with a supervised term can improve selection. Across 134 candidate families, teacher‑anchored selection reduces mean regret with very few labels, though the benefit diminishes after about 25 labels.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computer Vision
Sep 3

Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.

By Kabir Thayani
llmsefficiencymultimodal
More like this →
arXiv Machine Learning
Aug 20

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

GEAR is a two‑stage framework that distills tabular foundation models into lightweight MLP or tree‑based predictors for efficient CPU deployment. In the first stage, synthetic covariates are used as teacher‑query locations to train the student on soft TFM targets, expanding coverage beyond observed rows. The second stage re‑anchors the student to the target distribution using real labels and out‑of‑fold teacher predictions, preventing self‑labeling leakage and improving performance. Experiments on TALENT and TabArena show that GEAR‑distilled MLPs outperform supervised MLPs by up to 2.00 AUC points on binary tasks and 1.35 on multiclass tasks, and also outperform CatBoost, while dramatically reducing inference time and memory usage.

By Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun
llmsefficiency
More like this →
arXiv Machine Learning
Sep 7

Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy

The paper introduces an unsupervised method for selecting essential neurons in overparameterized neural networks by minimizing mapping entropy (ME), a metric that quantifies the loss of discriminatory power when neurons are discarded. ME-based selection relies solely on hidden-activation statistics and, in experiments, identifies minimal teacher-consistent representations in teacher‑student networks and coherent functional-class mappings in a non‑linear Gaussian process task. Subnetworks chosen by ME outperform random subsets of the same size, especially under strong compression, on both a Gaussian process task and translation‑augmented MNIST.

By Margherita Mele, Andrea Castagna, Roberto Menichetti, Raffaello Potestio, Alessandro Ingrosso
More like this →