The paper investigates how test‑time compute techniques—adaptation, aggregation, and context construction—can improve predictions of pretrained tabular foundation models (TFMs). It introduces DiagScale, a lightweight adaptation method that trains a minuscule fraction of parameters yet matches full fine‑tuning gains, and shows that selective aggregation of predictions from diverse preprocessing variants can reduce error, while broader context expansion offers inconsistent benefits. The study concludes that adaptation and selective aggregation consistently yield benchmark‑level improvements, but their combined use demands more computation, guiding strategy selection based on available resources.
By Kanghui Ning, Marin Bilo\v{s}, James T. Wilson, Yilang Zhang, Kashif Rasul, Dongjin Song, Anderson Schneider, Yuriy Nevmyvaka
The paper introduces RefineICL, an attention‑gated, feed‑forward‑network‑free framework that refines representations in situ for tabular foundation models. By using support labels to guide episode‑specific updates, the method transfers learned corrections to unlabeled queries without altering model parameters, achieving state‑of‑the‑art performance on AMLB29 and TabArena benchmarks. Experiments and internal interventions demonstrate that intermediate support updates are essential for constructing task‑specific predictors in context.
By Tian Zhou, Beverly Jin, Linxiao Yang, Xue Wang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun
arXiv:2610.19814v1 Announce Type: new
Abstract: Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while process...
By Wenwen Gong, Haotian Gu, Zhen Yang, Jiaying Ma, Hanqing Zhang, Zhongshen Wang, Jingjun Wang, Shouyi Xu, Wei Wang
The paper introduces Retro, a tabular foundation model that uses retrospective inference to revisit and recombine intermediate representations during inference. Retro employs Attention Residuals to select which past information to reuse and query-conditioned Gated Attention to shape the contextual update for each query. Experiments on TabArena, TALENT, and RelArena show Retro achieves top‑three performance and lies on the Pareto frontier, demonstrating that reusing intermediate representations can better exploit depth in tabular foundation models.
By Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng, Kun-Yang Yu, Jin-Hao Sheng, Guo Yu, Chonghan Liu, Zhi Zhou, Jun-Peng Jiang, Lan-Zhe Guo, Han-Jia Ye
arXiv:2608. 16429v1 Announce Type: new Abstract: Foundational models for tabular data have made significant progress in recent years, with TabICLv2 reporting state-of-the-art performance on several tabular classification tasks.
By Beimnet Bekele Guta
The paper shows that Tabular Foundation Models (TFMs) are highly overparameterized, with up to 96% of transformer blocks removable without significant loss in performance. Using a closed‑form, label‑free affine translator, the authors demonstrate that halving the depth of eight leading TFMs costs at most $0.002$ median AUROC, while a single‑block TabPFN‑3 remains competitive. Removing depth saves up to 1.5 B parameters, speeds GPU inference up to 6×, and reduces memory usage by 6× with negligible accuracy loss.
By Irene Cannistraci, Julia E. Vogt
The study shows that tabular foundation models (TFMs) can be dramatically simplified without losing performance. By replacing up to 96% of transformer blocks with a label‑free affine translator, researchers found that half‑depth models still outperform full‑depth TFMs and tuned GBDTs across 171 classification datasets. The simplification reduces parameters by up to 1.5 B, speeds GPU inference up to 6×, and cuts memory usage by 6× with minimal impact on AUROC.
arXiv:2605. 18383v2 Announce Type: replace Abstract: We present TabH2O, a foundation model for tabular data that performs classification and regression in a single forward pass via in-context learning.
By Pascal Pfeiffer, Dmitry Gordeev, Mathias M\"uller, Laura Fink, Joan Salv\`a Soler, Mark Landry, Branden Murray, Marcos V. Conde, Sri Satish Ambati
arXiv:2608. 01400v1 Announce Type: new Abstract: Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity.
By Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han, Valentin Thomas, Anthony L. Caterini
LoopICL is a transformer architecture that loops a single block to address tabular tasks. It separates parameter count from computational depth by using a cell stream for per‑cell features and a row stream for in‑context examples, refined via within‑column and cross‑column attention. During pre‑training, varying loop counts and a learned exit‑gate allow the model to adjust inference depth at test time, achieving competitive performance with TabICLv2 while using about 90% fewer parameters.
By Amir Rezaei Balef, Katharina Eggensperger
arXiv:2608. 10837v1 Announce Type: cross Abstract: The strong performance of foundation models for tabular tasks comes at substantial inference costs.
By Mykhailo Koshil, Matthias Feurer, Katharina Eggensperger
arXiv:2609.37989v1 Announce Type: new
Abstract: Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, ta...
By Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan, Sujay Sanghavi, Abhimanyu Das, Weihao Kong