GEAR is a two‑stage framework that distills tabular foundation models into lightweight MLP or tree‑based predictors for efficient CPU deployment. In the first stage, synthetic covariates are used as teacher‑query locations to train the student on soft TFM targets, expanding coverage beyond observed rows. The second stage re‑anchors the student to the target distribution using real labels and out‑of‑fold teacher predictions, preventing self‑labeling leakage and improving performance. Experiments on TALENT and TabArena show that GEAR‑distilled MLPs outperform supervised MLPs by up to 2.00 AUC points on binary tasks and 1.35 on multiclass tasks, and also outperform CatBoost, while dramatically reducing inference time and memory usage.
By Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun
The paper investigates active in-context learning for Tabular Foundation Models (TFMs), focusing on how different acquisition strategies—diversity, predictive uncertainty, and target-aware information gain—perform as more labeled data is added. It introduces Target-Aware Nearest Neighbor (TANN) and Expected Predictive Information Gain in Context (EPIC) methods, comparing them to classical rules across twenty benchmarks with two TFMs and gradient-boosted decision trees. Key findings include that diversity-based acquisition is most effective early on, target-aware nearest-neighbor labeling significantly improves performance, and EPIC achieves near full-data performance while labeling only 5–10% of the pool.
By Christian M. M. Frey
arXiv:2509. 09960v2 Announce Type: replace-cross Abstract: Synthetic tabular data generation is increasingly essential in machine learning, supporting downstream applications when real-world, high-quality tabular data is insufficient.
By Mingxuan Jiang, Keyang Chen, Yongxin Wang, Yongsheng Zhao, Ziyue Dai, Yicun Liu, Zeping Li, Qiuyang Zhang, Hongyi Nie, Hongbin Zhu, Sen Liu, Guangnan Ye, Hongfeng Chai
arXiv:2610.01143v1 Announce Type: cross
Abstract: Despite strong mean accuracy, tabular foundation models (TFMs) can perform poorly on underrepresented groups under subpopulation shift, where group p...
By Seonghwi Kim, Sung Ho Jo, Minwoo Chae
The paper presents a method for distilling tabular foundation models (TFMs) into lightweight, dataset‑specific students. By using the full labeled training set as teacher context and training students on both observed and synthetic queries, the authors achieve significant performance gains over traditional supervised models on TabArena and TALENT benchmarks. The distilled students also provide substantial inference speedups, reducing the cost of repeated inference.
By Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
The paper demonstrates that a tabular foundation model can achieve strong generalization using only a single real table for self‑supervised pre‑training, challenging the belief that large synthetic or real datasets are necessary. By systematically pre‑training and evaluating across diverse benchmarks, the authors show that the number and quality of tasks that can be derived from a dataset are critical for downstream performance. This finding suggests that carefully constructed task sets from limited data can enable effective transfer learning in tabular models.
By Junwei Ma, Nour Shaheen, Alex Labach, Amine Mhedhbi, Frank Hutter, Anthony L. Caterini, Valentin Thomas
The paper introduces Retro, a tabular foundation model that uses retrospective inference to revisit and recombine intermediate representations during inference. Retro employs Attention Residuals to select which past information to reuse and query-conditioned Gated Attention to shape the contextual update for each query. Experiments on TabArena, TALENT, and RelArena show Retro achieves top‑three performance and lies on the Pareto frontier, demonstrating that reusing intermediate representations can better exploit depth in tabular foundation models.
By Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng, Kun-Yang Yu, Jin-Hao Sheng, Guo Yu, Chonghan Liu, Zhi Zhou, Jun-Peng Jiang, Lan-Zhe Guo, Han-Jia Ye
TabMF is a framework that repurposes pretrained tabular foundation models (TFMs) as encoders for supervised datasets, adding a dataset-level readout and fine‑tuning via contrastive post‑training on synthetic views. The authors instantiate TabMF on TabICLv2 and TabPFN‑2.5 and evaluate the resulting representations on six tasks involving unseen real datasets. The approach achieves the best aggregate performance among both learned and classical dataset representations, excelling in dataset retrieval, compact benchmark construction, and distribution‑shift severity assessment.
By Rafiq Kamel, Nicolas Atienza, Aziz Fouch\'e, Nicola Cancedda, Josif Grabocka
The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table. It finds that a table’s usefulness is largely determined by its number of features rather than instances, and that fine‑grained column‑level preprocessing improves downstream performance while dataset‑level filtering does not. The authors propose that tabular in‑context generalization is primarily retrieval‑based, with models learning to identify and aggregate relevant examples from the provided context.
By Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini
The paper investigates how synthetic pretraining priors used in tabular foundation models (TFMs) influence downstream performance. By reconstructing the synthetic data generators of four TFMs and comparing their generated tasks to two popular tabular benchmarks using structural descriptors, the authors measure structural coverage and normalized density. They find that some generators provide broader and denser support for benchmark tasks, and that stronger synthetic-to-benchmark support generally correlates with better model performance.
By He Zhao, Ryan Thompson, Daniel M. Steinberg, Ashfaqur Rahman, Edwin V. Bonilla, Cheng Soon Ong
arXiv:2606. 29241v1 Announce Type: new Abstract: Data-generating priors are a central component of tabular foundation models because they define the task distribution used during pretraining.
By Zeynep T\"urkmen, K\"ur\c{s}at Kaya, Alexander Pfefferle, Frank Hutter
The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table, rather than large synthetic or real datasets. It finds that a table’s usefulness for downstream tasks is mainly determined by the number of features, not instances, and that fine‑grained column‑level preprocessing improves performance while dataset‑level filtering does not. The authors propose a task‑centric, retrieval‑based view of in‑context generalization, suggesting that effective TFMs identify and aggregate relevant examples from the provided context.