arXiv Machine Learning
6d ago
Small-Scale Experiments: Are We There Yet?
arXiv:2608. 11859v1 Announce Type: new Abstract: Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver.
Small-Scale Experiments: Are We There Yet?
Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales (starting at 4M parameters) and concluded that sizable models cannot be avoided.
Related stories
arXiv AI
Jul 3
$\mu$pscaling small models: Principled warm starts and hyperparameter transfer
arXiv:2602. 10545v2 Announce Type: replace-cross Abstract: Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets.
arXiv AI
Jun 9
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
arXiv:2606. 07616v1 Announce Type: cross Abstract: Scaling laws provide a fundamental framework for understanding the performance of Language Models (LMs), yet deriving them requires prohibitively expensive evaluations across thousands of checkpoints or millions of inference samples.
arXiv Machine Learning
6d ago
Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection
arXiv:2604. 22753v2 Announce Type: replace Abstract: Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions.
arXiv Machine Learning
Jun 2
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.
arXiv Machine Learning
Jun 26
How Good Can Linear Models Be for Time-Series Forecasting?
arXiv:2606. 27282v1 Announce Type: new Abstract: Time-series forecasting research has been moving steadily toward larger architectures, from specialized transformers to general-purpose foundation models, on the assumption that capacity is what unlocks accuracy.
arXiv Machine Learning
Aug 12
UniScale: Synergistic Entire Space Data and Model Scaling for Search Ranking
arXiv:2603. 24226v4 Announce Type: replace-cross Abstract: Recent advances in Large Language Models (LLMs) have inspired a surge of scaling research in industrial search, advertising, and recommendation systems.
Hugging Face Trending Papers
Jun 25
How Good Can Linear Models Be for Time-Series Forecasting?
Time-series forecasting research has been moving steadily toward larger architectures, from specialized transformers to general-purpose foundation models, on the assumption that capacity is what unlocks accuracy. We take the opposite position: most of the gap can be closed at far lower cost by tuning preprocessing rather than scaling models.
Hugging Face Trending Papers
Jul 25
Predict before you train: Scaling Laws for particle physics foundation models
The largest machine learning models in particle physics are also the most expensive to train, yet the return on scaling a given architecture cannot be estimated before that compute is spent. Scaling laws have been fit for jets, but none has yet been shown to predict the performance of models it was not fit on.
arXiv AI
Jun 11
Unifying Learning Dynamics and Generalization in Transformers Scaling Law
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
arXiv Machine Learning
Jun 2
From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction
arXiv:2511. 12081v2 Announce Type: replace-cross Abstract: Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns -- a stark contrast to the {predictable scaling laws} seen in large language models (LLMs).
arXiv Machine Learning
Jun 25
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.