← Back to all news
arXiv Computation and Language

Variable names provide priors: semantic grounding enables data-efficient numerical prediction by large language models

The study demonstrates that large language models (LLMs) can perform numerical prediction on tabular data by leveraging semantic grounding—matching variable names to the values they describe—without task-specific training. Across 20 datasets from nine scientific domains, five LLMs matched or outperformed seven supervised machine learning models, especially when training examples were scarce. Controlled experiments with physical formulas reveal that LLMs combine domain knowledge (as a prior) with in-context examples (likelihood) in a Bayesian‑like manner, adjusting predictions as contradictory examples accumulate.

Related stories

arXiv AI
Oct 16

Language Models Struggle to Use Representations Learned In-Context

The article examines whether large language models (LLMs) can effectively use representations they learn in-context to perform downstream tasks. Experiments with open‑weight LLMs on next‑token prediction and a new adaptive world modeling task reveal that these models struggle to deploy novel semantics defined in-context, even when such semantics are encoded in their latent representations. The study also shows that state‑of‑the‑art closed‑source reasoning models similarly fail to reliably leverage in‑context patterns, highlighting a gap in flexible representation usage.

By Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova
llmsbenchmarks
More like this →
arXiv Machine Learning
Aug 19

Understanding the Surprising Generalization Properties of Tabular Foundation Models

The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table. It finds that a table’s usefulness is largely determined by its number of features rather than instances, and that fine‑grained column‑level preprocessing improves downstream performance while dataset‑level filtering does not. The authors propose that tabular in‑context generalization is primarily retrieval‑based, with models learning to identify and aggregate relevant examples from the provided context.

By Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini
llms
More like this →
arXiv Machine Learning
Sep 30

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...

By Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko
llmsbenchmarks
More like this →
Hugging Face Trending Papers
Aug 18

Understanding the Surprising Generalization Properties of Tabular Foundation Models

The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table, rather than large synthetic or real datasets. It finds that a table’s usefulness for downstream tasks is mainly determined by the number of features, not instances, and that fine‑grained column‑level preprocessing improves performance while dataset‑level filtering does not. The authors propose a task‑centric, retrieval‑based view of in‑context generalization, suggesting that effective TFMs identify and aggregate relevant examples from the provided context.

llms
More like this →