arXiv Machine Learning
When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 8
Thinking in Depth: Retrospective Inference for Tabular Foundation Models
The paper introduces Retro, a tabular foundation model that uses retrospective inference to revisit and recombine intermediate representations during inference. Retro employs Attention Residuals to select which past information to reuse and query-conditioned Gated Attention to shape the contextual update for each query. Experiments on TabArena, TALENT, and RelArena show Retro achieves top‑three performance and lies on the Pareto frontier, demonstrating that reusing intermediate representations can better exploit depth in tabular foundation models.
arXiv AI
Oct 12
Tabular Foundation Models Are Effectively Shallow
The paper shows that Tabular Foundation Models (TFMs) are highly overparameterized, with up to 96% of transformer blocks removable without significant loss in performance. Using a closed‑form, label‑free affine translator, the authors demonstrate that halving the depth of eight leading TFMs costs at most $0.002$ median AUROC, while a single‑block TabPFN‑3 remains competitive. Removing depth saves up to 1.5 B parameters, speeds GPU inference up to 6×, and reduces memory usage by 6× with negligible accuracy loss.
arXiv Machine Learning
Oct 2
VIP-COP: Context Optimization for Tabular Foundation Models
arXiv:2605.12904v2 Announce Type: replace Abstract: Tabular foundation models (TFMs) have emerged as a powerful paradigm for in-context learning on structured data, enabling direct prediction on new...
arXiv Machine Learning
Sep 30
Architecture Alignment With Sparse Priors in Tabular Foundation Models
arXiv:2609.36883v1 Announce Type: new Abstract: Tabular foundation models (TFMs) are increasingly popular because they deliver strong predictions on new datasets through in-context learning, without...
arXiv Machine Learning
4d ago
OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
OrionMoE-v1.0 is a compact tabular foundation model that uses a dual‑axis Transformer and a sparse mixture‑of‑experts architecture to perform in‑context learning on labeled support tables. It selectively activates experts along the feature axis, guided by both individual feature representations and overall dataset context, enabling adaptive computation across diverse tabular problems. Pretrained on synthetic classification tasks, it achieves competitive performance on multiple benchmarks, especially on small‑context, high‑dimensional, and class‑imbalanced datasets.
arXiv AI
Sep 10
From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models
The paper investigates how synthetic pretraining priors used in tabular foundation models (TFMs) influence downstream performance. By reconstructing the synthetic data generators of four TFMs and comparing their generated tasks to two popular tabular benchmarks using structural descriptors, the authors measure structural coverage and normalized density. They find that some generators provide broader and denser support for benchmark tasks, and that stronger synthetic-to-benchmark support generally correlates with better model performance.