arXiv Statistics ML
Confidence Is Not a Promise: A Reference Audit of Typed Decision Models
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Computation and Language
Oct 7
Readout Stability in Prefill-Only Decision Models:Zero-Label Prediction and Inference-Time Compute Allocation
Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.
arXiv Computer Vision
1d ago
How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop
The paper introduces a denoised counterfactual protocol to accurately measure the selectable headroom of world action models (WAMs) and uses these measurements to build a fail‑gated selector. Across four WAMs and two benchmarks, the study shows that the standard estimate overstates selectable headroom, that structured action perturbations can add up to 16 points of headroom, and that imagined futures retain candidate‑outcome information without loss. The selector improves closed‑loop task success from 76.9 % to 81.6 % on held‑out training episodes and from 62.6 % to 69.8 % on unseen tasks.
arXiv Computation and Language
1d ago
3, 7, 14, 17, 42: Instruction Tuning Installs and Sharpens Number-Choice Heuristics in Language Models
The study measures the exact probability that language models assign to each integer from 1 to N when prompted for a random number, eliminating sampling noise by summing over all spellings. Across 18 open‑weight models and 29 decoding settings, instruction tuning consistently makes the distribution less uniform, introducing a round‑number avoidance heuristic and sharpening endpoint avoidance. While tuned models rarely output the extreme values 1 or N and largely avoid multiples of 10, they do not converge on a single favorite integer across models.
arXiv Machine Learning
Jul 1
Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models
arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.
arXiv Machine Learning
Aug 19
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
The paper evaluates how three large mixture‑of‑experts models (Alibaba, OpenAI, NVIDIA) can be fine‑tuned to reason in a low‑resource language, specifically Greek. Accuracy metrics show little change, but the authors uncover significant qualitative improvements: after supervised fine‑tuning, models reason in Greek on ~98% of items, with better grammaticality and retained general ability. Reinforcement learning with pre‑registered rewards further eliminates reasoning‑channel leaks and format skips, while the Greek‑reasoning habit remains robust to an accuracy‑only gradient.
arXiv AI
Sep 30
Evaluating and Benchmarking the System One Model Jev
arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...