TabReliBench evaluates tabular foundation models (TFMs) on reliability metrics beyond predictive accuracy, including point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. The benchmark reveals that TFMs’ performance rankings differ from those in TabArena and vary across reliability measures. While regression TFMs often achieve near‑nominal marginal coverage with narrow intervals, this does not guarantee valid conditional coverage, and ensembling or conformal post‑processing does not consistently improve calibration. The study underscores the need for multidimensional evaluation when selecting reliable TFMs.
By Tianqi Zhao, Jia Guo, Yan Shuo Tan, Qiong Zhang
arXiv:2606. 08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts.
By Bitya Neuhof, Yuval Benjamini
arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.
By Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang
The paper argues that typical tabular machine learning benchmarks, which aggregate results by averaging scores or ranks, can hide which models are essential for achieving the best performance on specific datasets. It proposes evaluating models against a data‑centric peak performance frontier, classifying them as irreplaceable, sufficient, redundant, or fallible based on their position relative to other models. Applying this to the TabArena benchmark shows that common aggregation metrics mainly capture consistency and failure avoidance, but fail to reflect dataset‑specific strengths, leading to a misalignment between aggregate rewards and true model utility.
By Andrej Tschalzev, Stefan L\"udtke, Heiner Stuckenschmidt, Christian Bartelt
Tabular machine learning benchmarks typically summarize performance by averaging scores, ranks, or pairwise wins across datasets. Such aggregates are useful for selecting robust default models, but th...
Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features.