The study shows that tabular foundation models (TFMs) can be dramatically simplified without losing performance. By replacing up to 96% of transformer blocks with a label‑free affine translator, researchers found that half‑depth models still outperform full‑depth TFMs and tuned GBDTs across 171 classification datasets. The simplification reduces parameters by up to 1.5 B, speeds GPU inference up to 6×, and cuts memory usage by 6× with minimal impact on AUROC.
arXiv:2608. 10837v1 Announce Type: cross Abstract: The strong performance of foundation models for tabular tasks comes at substantial inference costs.
By Mykhailo Koshil, Matthias Feurer, Katharina Eggensperger
arXiv:2607. 27546v1 Announce Type: new Abstract: Tabular Foundation Models, such as TabPFN, have received a large amount of recent attention due to their performance on in-context tabular machine learning tasks, which often exceeds classical baselines.
By Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon, Anna Leontjeva, Simon Lucey
arXiv:2606. 04485v1 Announce Type: new Abstract: Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states.
By Yuanrui Wang, Xingxuan Zhang, Han Yu, Mingchao Ming, Gang Ren, Hao Yuan, Li Mao, Yunjia Zhang, Chun Yuan, Peng Cui
Tydra is a hybrid Transformer‑State Space Model that interleaves attention and SSM layers for tabular in‑context learning. It achieves a 30% reduction in inference time compared to the Transformer‑only TabPFN while preserving most of its predictive performance. On 30 OpenML datasets, Tydra also outperforms a Hydra model that is roughly ten times larger, demonstrating that hybrid architectures can balance accuracy and efficiency for tabular foundation models.
By Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian Kersting
The paper investigates how test‑time compute techniques—adaptation, aggregation, and context construction—can improve predictions of pretrained tabular foundation models (TFMs). It introduces DiagScale, a lightweight adaptation method that trains a minuscule fraction of parameters yet matches full fine‑tuning gains, and shows that selective aggregation of predictions from diverse preprocessing variants can reduce error, while broader context expansion offers inconsistent benefits. The study concludes that adaptation and selective aggregation consistently yield benchmark‑level improvements, but their combined use demands more computation, guiding strategy selection based on available resources.
By Kanghui Ning, Marin Bilo\v{s}, James T. Wilson, Yilang Zhang, Kashif Rasul, Dongjin Song, Anderson Schneider, Yuriy Nevmyvaka