← Back to all news
arXiv Machine Learning By Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

Read the original on arXiv Machine Learning →

GEAR is a two‑stage framework that distills tabular foundation models into lightweight MLP or tree‑based predictors for efficient CPU deployment. In the first stage, synthetic covariates are used as teacher‑query locations to train the student on soft TFM targets, expanding coverage beyond observed rows. The second stage re‑anchors the student to the target distribution using real labels and out‑of‑fold teacher predictions, preventing self‑labeling leakage and improving performance. Experiments on TALENT and TabArena show that GEAR‑distilled MLPs outperform supervised MLPs by up to 2.00 AUC points on binary tasks and 1.35 on multiclass tasks, and also outperform CatBoost, while dramatically reducing inference time and memory usage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 3

Distillation of Tabular Foundation Models into Efficient Predictors

The paper presents a method for distilling tabular foundation models (TFMs) into lightweight, dataset‑specific students. By using the full labeled training set as teacher context and training students on both observed and synthetic queries, the authors achieve significant performance gains over traditional supervised models on TabArena and TALENT benchmarks. The distilled students also provide substantial inference speedups, reducing the cost of repeated inference.

By Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
llmsefficiency
More like this →
arXiv Machine Learning
Sep 24

What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates

The paper introduces RefineICL, an attention‑gated, feed‑forward‑network‑free framework that refines representations in situ for tabular foundation models. By using support labels to guide episode‑specific updates, the method transfers learned corrections to unlabeled queries without altering model parameters, achieving state‑of‑the‑art performance on AMLB29 and TabArena benchmarks. Experiments and internal interventions demonstrate that intermediate support updates are essential for constructing task‑specific predictors in context.

By Tian Zhou, Beverly Jin, Linxiao Yang, Xue Wang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun
benchmarks
More like this →
arXiv Machine Learning
Aug 14

TabH2O: A Unified Foundation Model for Tabular Prediction

arXiv:2605. 18383v2 Announce Type: replace Abstract: We present TabH2O, a foundation model for tabular data that performs classification and regression in a single forward pass via in-context learning.

By Pascal Pfeiffer, Dmitry Gordeev, Mathias M\"uller, Laura Fink, Joan Salv\`a Soler, Mark Landry, Branden Murray, Marcos V. Conde, Sri Satish Ambati
llmsbenchmarks
More like this →
arXiv AI
Aug 24

MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation

MentorPulse introduces a method for refreshing cross‑model latent guidance during long‑form generation. By compressing mentor states into a capped slot memory and updating it incrementally as new tokens are generated, the approach keeps the guidance signal fresh without resetting the student’s KV cache. Across thirteen datasets, MentorPulse closes more than half of the mentor‑student performance gap, outperforming existing techniques such as C2C, T2T, and LoRA, especially on long outputs.

By Ziwu Liu, Guozhong Li, Chen Qiu, Weiyang Kong, Panos Kalnis
fine-tuningefficiency
More like this →