arXiv Machine Learning
Jun 2
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.
Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
arXiv:2605. 09724v2 Announce Type: replace Abstract: Existing accounts of grokking explain the phenomena in terms of mechanistic frameworks such as circuit efficiency or lazy-to-rich transitions.
Related stories
Hugging Face Trending Papers
Oct 6
How Learning Governs Unlearning across the Memorization-Generalization Spectrum
While unlearning seeks to negate undesired capabilities acquired through learning, little research has examined how the way models learn shapes their subsequent unlearning. In this paper, we investiga...
arXiv AI
6d ago
How Learning Governs Unlearning across the Memorization-Generalization Spectrum
arXiv:2610.08577v1 Announce Type: cross Abstract: While unlearning seeks to negate undesired capabilities acquired through learning, little research has examined how the way models learn shapes their...
arXiv AI
Jul 7
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
arXiv:2607. 05104v1 Announce Type: cross Abstract: Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs.
arXiv Machine Learning
Sep 11
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
arXiv AI
Aug 17
Capacity-Dependent Effects of Data Selection for Reasoning
arXiv:2608. 13721v1 Announce Type: cross Abstract: In reasoning supervised fine-tuning, candidate responses for the same instruction can differ substantially in how well they match the student's current distribution.
arXiv AI
Oct 5
Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
arXiv:2512. 19905v3 Announce Type: replace-cross Abstract: Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time.
Hugging Face Trending Papers
Jul 15
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately. What happens at the extreme of this spectrum, when the architecture's expressible function class collapses to a finite-dimensional algebraic variety?
arXiv Machine Learning
Sep 24
Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models
The paper investigates how large language models (LLMs) balance capability and efficiency when using Chain-of-Thought reasoning on arithmetic and algorithmic tasks. It finds that while larger models solve more problems correctly, the improvement follows an exponential decay that slows with scale, indicating diminishing returns. Additionally, the length of reasoning output grows with problem size but does not improve with larger models, suggesting efficiency does not benefit from scaling.
arXiv Machine Learning
Jul 16
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
arXiv:2607. 13749v1 Announce Type: new Abstract: Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately.
arXiv Machine Learning
Oct 5
Capability Scaling-Down Laws for LLM Compression
The paper presents a systematic study of how different compression techniques—pruning, quantization, and distillation—affect the capabilities of large language models (LLMs) in tasks such as mathematics, code generation, and question answering. It introduces a framework that measures capability loss and relates it to factors like model size, training stage, and compression settings, yielding simple predictive relations that generalize across unseen configurations. The authors demonstrate that sharing density responses across pruning levels can dramatically reduce the number of measurements needed, and that their predictive models closely match regression results while offering efficient decision guidance for compression method selection.
arXiv AI
Aug 13
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
arXiv:2608. 12307v1 Announce Type: cross Abstract: Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods.