arXiv AI
Jun 9
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
Hinted Scaling Laws
The paper introduces hinted scaling laws, which incorporate privileged information to transform a benchmark into a continuum of easier versions, ensuring even small models achieve non‑zero accuracy. This approach yields smoother, more predictable accuracy trends, allowing accurate extrapolation to large models even when standard scaling laws fail. Across multiple pretraining ladders and model families, hinted scaling laws dramatically reduce forecasting errors and the number of test examples needed to distinguish model performance.
Related stories
arXiv Machine Learning
Oct 19
Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction
Predictable Scaling (PS) is a framework that uses small‑scale training proxies to forecast full‑scale model performance, reducing the need for expensive GPU‑hourly runs. By applying composable scaling functions and curve fitting with uncertainty estimates, PS can rank candidate changes accurately across different hypothesis types. Experiments on two ranking models show PS achieves 79–88% compute savings while maintaining correct ranking of actionable hypotheses.
arXiv AI
Jun 11
Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation
arXiv:2606. 12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.
arXiv AI
Jun 9
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
arXiv:2606. 07616v1 Announce Type: cross Abstract: Scaling laws provide a fundamental framework for understanding the performance of Language Models (LMs), yet deriving them requires prohibitively expensive evaluations across thousands of checkpoints or millions of inference samples.
arXiv Computation and Language
Sep 10
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.
arXiv AI
Jul 31
Predict before you train: Scaling Laws for particle physics foundation models
arXiv:2607. 23377v1 Announce Type: cross Abstract: The largest machine learning models in particle physics are also the most expensive to train, yet the return on scaling a given architecture cannot be estimated before that compute is spent.
arXiv Statistics ML
Oct 28
Confidence Is Not a Promise: A Reference Audit of Typed Decision Models
The paper evaluates how well typed decision models, which output probability distributions, actually provide reliable guarantees. By comparing open-weight Laya and Jev 1.13 across 190 evaluations, the study finds that point‑estimate thresholds frequently break promised accuracy, while a Beta lower‑bound rule and Learn‑then‑Test approach maintain guarantees more consistently. The work also explores calibration across languages, the impact of noise, data requirements for fine‑tuning, and the limitations of confidence in predicting outcomes or providing faithful explanations.
arXiv AI
Jul 29
Bridging Compute- and Data-Optimal Pretraining
arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
arXiv AI
Sep 7
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
The paper demonstrates that layer dropout, also known as stochastic depth, can be effectively used in state‑of‑the‑art large language model (LLM) training. By optimizing the layer distribution, schedule, and optimizer settings, the authors show that layer dropout can reduce training loss while saving up to 25 % of training FLOPs. Additionally, layer dropout enables post‑training optimizations such as early exit and self‑speculative decoding, achieving up to 1.5× inference speedup with negligible accuracy loss across models ranging from 271 M to 8.2 B parameters and datasets up to 160 B tokens.
arXiv Machine Learning
Sep 23
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.
arXiv Computation and Language
Oct 7
Readout Stability in Prefill-Only Decision Models:Zero-Label Prediction and Inference-Time Compute Allocation
Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.
arXiv Machine Learning
Aug 11
Efficient Test-Time Scaling for LLM-based Time Series Forecasting
arXiv:2608. 08675v1 Announce Type: new Abstract: Long-term time series forecasting benefits from preserving global structure such as trends and seasonality.