← Back to all news
arXiv Machine Learning By Kanghui Ning, Marin Bilo\v{s}, James T. Wilson, Yilang Zhang, Kashif Rasul, Dongjin Song, Anderson Schneider, Yuriy Nevmyvaka

Test-Time Compute for Tabular Foundation Models: Mechanisms, Gains, and Limits

Read the original on arXiv Machine Learning →

The paper investigates how test‑time compute techniques—adaptation, aggregation, and context construction—can improve predictions of pretrained tabular foundation models (TFMs). It introduces DiagScale, a lightweight adaptation method that trains a minuscule fraction of parameters yet matches full fine‑tuning gains, and shows that selective aggregation of predictions from diverse preprocessing variants can reduce error, while broader context expansion offers inconsistent benefits. The study concludes that adaptation and selective aggregation consistently yield benchmark‑level improvements, but their combined use demands more computation, guiding strategy selection based on available resources.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 24

What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates

The paper introduces RefineICL, an attention‑gated, feed‑forward‑network‑free framework that refines representations in situ for tabular foundation models. By using support labels to guide episode‑specific updates, the method transfers learned corrections to unlabeled queries without altering model parameters, achieving state‑of‑the‑art performance on AMLB29 and TabArena benchmarks. Experiments and internal interventions demonstrate that intermediate support updates are essential for constructing task‑specific predictors in context.

By Tian Zhou, Beverly Jin, Linxiao Yang, Xue Wang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun
benchmarks
More like this →
arXiv Machine Learning
Sep 29

Benchmarking Attention for Tabular Foundation Models

The paper introduces a reproducible benchmark for evaluating attention mechanisms in tabular foundation models, focusing on the distinct row and column attention patterns that differ from language model attention. It compares several backends—Torch SDPA, FlashAttention variants, vLLM, and SageAttention—across realistic tabular shapes on A100, H100, and B200 GPUs, revealing that optimal backend choice varies by attention type, hardware, and model specifics. The study finds FlashAttention generally performs best, but CuDNN can outperform it for column attention on longer sequences, while SageAttention excels for large row sequences beyond 16k rows.

By Maximilian Schambach, Clemens Biehl, Sam Thelin
llmsragefficiencybenchmarks
More like this →
arXiv Machine Learning
Aug 19

Understanding the Surprising Generalization Properties of Tabular Foundation Models

The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table. It finds that a table’s usefulness is largely determined by its number of features rather than instances, and that fine‑grained column‑level preprocessing improves downstream performance while dataset‑level filtering does not. The authors propose that tabular in‑context generalization is primarily retrieval‑based, with models learning to identify and aggregate relevant examples from the provided context.

By Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini
llms
More like this →