arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound.
The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.
arXiv:2611.05539v1 Announce Type: new Abstract: LLMs can repair noisy OCR transcripts of historical documents, but they also change text that was already correct. We propose EditVeto, a veto gate pla...
arXiv:2603. 11784v2 Announce Type: replace Abstract: As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed.
The paper introduces a novel patching strategy for byte‑level language models, called novelty patching, which starts a new patch when upcoming bytes are unlikely to repeat recent context, inspired by LZ77 compression. Experiments on a 1.9 GB math corpus show that novelty patching improves final‑answer accuracy from 47 % to 72 % at 12 M parameters and raises computed‑result accuracy from 21 % to 57 % at 50 M parameters, while having limited impact on prose and source code. The study also analyzes patch boundary distributions and reports several failed approaches, such as typed uncertainty and context‑aware novelty prediction. whyItMatters":"The findings demonstrate that a simple, data‑driven patching rule can significantly enhance the performance of byte‑level models on tasks involving computed results, offering a practical improvement for large language models that process raw text without explicit labels."
The paper proposes a framework for studying how alignment difficulty scales with model size, treating alignment as a family of measurable scaling relations for different risk categories. It introduces operational definitions of burden, distinguishes observed, audited, and true alignment, and proves that the largest exponent among corrected risks determines long‑run behavior. The authors preregistered and applied the protocol to public data, finding scaling exponents for attack success, truthfulness, dispositions, sycophancy, and a backdoor, and released interactive browser games to illustrate the laws.
arXiv:2610.15093v1 Announce Type: cross Abstract: AI coding agents operate in iterative loops: they modify code, run tools, inspect the results, and decide what to do next. We call the design of this...
arXiv:2606. 06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus.
arXiv:2609.00378v1 Announce Type: new Abstract: Large language models pay a well-documented tax on non-English text: the same content costs several times more tokens, and because attention is quadrat...
The paper shows that long‑context drafting can be performed with short‑context drafters that are frozen and run beyond their training length, using two inference‑time mechanisms to keep their cost bounded. By bounding the attention cache and re‑basing rotary position indices, the authors demonstrate that the speedup achieved with a short‑context drafter persists up to two million tokens across multiple target‑drafter pairs and corpora. They prove that when a common index shift leaves the drafter’s computation unchanged, the acceptance gap can be tightly bounded, ensuring consistent performance.
The paper introduces Power‑Law Entropy Search (PLES), a computational‑cost‑aware acquisition function that uses multi‑fidelity Bayesian optimization to efficiently estimate optimal hyperparameter scaling laws for large language model training. PLES focuses on reducing the overall uncertainty of scaling law estimates rather than optimizing a single objective, selecting configurations that maximize uncertainty reduction per unit computational cost. Experiments on synthetic benchmarks, surrogate models, and real LLM pre‑training runs show that PLES converges to accurate scaling laws using less than one‑tenth of the computational budget required by conventional grid search and other baselines.