arXiv:2608.21653v1 Announce Type: cross Abstract: Multi-task learning often combines losses that span several orders of magnitude, causing homoscedastic uncertainty weighting to degrade severely. We...
Predictable Scaling (PS) is a framework that uses small‑scale training proxies to forecast full‑scale model performance, reducing the need for expensive GPU‑hourly runs. By applying composable scaling functions and curve fitting with uncertainty estimates, PS can rank candidate changes accurately across different hypothesis types. Experiments on two ranking models show PS achieves 79–88% compute savings while maintaining correct ranking of actionable hypotheses.
arXiv:2605. 31244v2 Announce Type: replace Abstract: Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance.
arXiv:2610.07324v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) are trained on large collections of time series datasets that span various morphologies and domains. This setti...
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
Existing machine learning (ML) scaling laws relate predictive loss to compute, model parameters, and data. However, as models are increasingly deployed through agentic harnesses, loss alone is insuffi...
The paper introduces the Capability-Driven Multimodal Scaling Law, a cross-family framework that predicts vision-language model (VLM) benchmark accuracy from a low-dimensional textual capability score extracted via PCA. By training over 150 VLMs on 34 large language models across seven families, the authors demonstrate that the law accurately extrapolates transfer rates from 8B to 72B‑parameter backbones, predicts full training trajectories, and generalizes to unseen model families. The study also reveals actionable insights, such as certain textual benchmarks negatively correlating with multimodal performance and base LLMs outperforming instruction-tuned counterparts as VLM backbones due to higher absorption rates.
The paper introduces a capability manifold, a multidimensional framework that maps downstream capabilities—such as reasoning, retrieval, planning, and adaptation—to pre‑training, post‑training, and test‑time resources via bounded scaling functions. It provides analytical Jacobians to quantify how sensitive each capability is to changes in resources and their interactions. By embedding existing Kaplan‑ and Chinchilla‑type scaling laws and test‑time compute into this manifold, the authors demonstrate that these scaling relationships can be unified as trajectories on a common capability manifold.
arXiv:2610.18227v1 Announce Type: cross Abstract: Domain-specific curvature is an appealing idea for domain generalization (DG), but curvature learning can denote substantially different computations...
The paper proposes a framework to reduce the cost of constructing scaling laws for large foundation models by treating data collection as a Bayesian optimization problem. It shows that expanding the compute budget progressively and augmenting observed configurations with surrogate-fantasized evaluations can recover a broad experimental grid, enabling accurate scaling law fitting without training every configuration. This approach can achieve computational savings of up to 10–100× compared to a full dense grid.
arXiv:2605.28940v2 Announce Type: replace-cross Abstract: Recently observed empirical scaling laws describe the performance of foundation-type models as three independent key quantities -- dataset si...
arXiv:2605. 29283v2 Announce Type: replace-cross Abstract: Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single average score under a fixed training distribution.