arXiv:2610.07804v1 Announce Type: new Abstract: Prior Fitted Networks (PFNs) such as TabPFN now rival established statistical procedures across prediction and estimation tasks. A natural explanation...
arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?
arXiv:2512. 24780v2 Announce Type: replace Abstract: Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking.
arXiv:2607. 02681v1 Announce Type: cross Abstract: Integrating information across related tasks can improve estimation and prediction in transfer, multi-task, and federated learning, but contamination and heterogeneity make robust borrowing challenging.
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
The paper introduces the Neural Function Minimizer (NFM), an iterative neural model that predicts the global minimum of a noisy one‑dimensional function from only twenty samples. It compares the NFM to two Set Transformers and classical zero‑query estimators, finding that all learned models outperform classical methods on held‑out data, with the NFM achieving the lowest error relative to the Gaussian‑process posterior argmin. The study also examines how different estimator outputs affect tie‑breaking between equally deep valleys and plans to release a benchmark and self‑checking harness.
arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.
arXiv:2605. 08475v3 Announce Type: replace-cross Abstract: In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass.
The paper introduces a transformer-based framework that learns an in‑context score operator for diffusion models, allowing the model to perform reverse‑diffusion sampling on unseen target distributions specified by a small empirical context set. By representing the task‑dependent score as a Gaussian‑kernel conditional mean and approximating it with a transformer‑based query‑dependent attention architecture, the method can adapt to new tasks without retraining or parameter updates. The authors provide theoretical generalization‑error bounds and evaluate the approach on scientific and visual datasets, noting strong performance on simple task families and limitations as task complexity grows.
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv:2609.37535v1 Announce Type: new Abstract: In the softmax output layer, a rare token receives a small positive logit gradient on most steps and a much larger negative gradient on the few steps w...