arXiv Machine Learning June 19, 2026 By Tiexin Ding

Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

Read the original on arXiv Machine Learning →

arXiv:2606. 19367v1 Announce Type: new Abstract: Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter $\lambda$ grows, overshoots, and then relaxes during AdamW training.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 28

Scale Weight Decay and Train Better

arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.

By Anuj Apte

safety

arXiv AI

Jul 14

Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.

By Ethan Smith

llms

arXiv Machine Learning

Aug 5

Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.

By Ravi Satya Durga Prasad Yenugula

llms

arXiv Machine Learning

Jul 24

Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat

arXiv:2607. 20552v1 Announce Type: new Abstract: Grokking -- the delayed generalization of neural networks long after they have memorized their training data -- wastes thousands of training epochs and is notoriously unpredictable.

By Chitraansh Pandey

llms

arXiv AI

Jun 2

Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling

arXiv:2605. 18838v3 Announce Type: replace-cross Abstract: Scaling laws predict loss from compute but not how capabilities interact.

By Adil Amin

llms efficiency benchmarks safety

arXiv Machine Learning

Jun 8

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

arXiv:2606. 06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus.

By Zhiwei Xu, Shihao Wu, Hanseul Cho, Wei Hu, Yixin Wang

llms diffusion

Related stories

Scale Weight Decay and Train Better

Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat

Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws