arXiv Machine Learning

Information Allocation Dynamics in Neural Network Optimization

arXiv:2607. 07156v1 Announce Type: new Abstract: Different optimizers have different update biases, but these biases are usually implicit.

arXiv Machine Learning

Jul 13

The Anatomy of Implicit Bias: Information Allocation in Neural Network Training

arXiv:2607. 07156v2 Announce Type: replace Abstract: Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry.

By Zhang Gongyue, Wang Zhiyong, Liu Donghan, Ren Weihong, Sheng Yixuan, Liu Honghai

safety

arXiv AI

Jul 14

Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.

By Ethan Smith

llms

arXiv Machine Learning

Jun 4

When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks

arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.

By Berk Tinaz, Changzhi Xie, Mahdi Soltanolkotabi

safety

arXiv Machine Learning

Jul 7

Target-Guided Selective Reweighting for Physics-Informed Neural Network Inverse Problems: A Transfer Learning Approach

arXiv:2607. 05271v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) encounter ill-posed optimization, loss competition, and parameter compensation in partial differential equation (PDE) inverse problems.

By Qian Hu, Bin Fan, Yao Xiao, Zhicheng Lin, Meixin Xiong

diffusion fine-tuning efficiency benchmarks

arXiv AI

Jul 10

Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization

arXiv:2603. 00910v2 Announce Type: replace-cross Abstract: Layer-wise capacity in large language models is highly non-uniform: some layers contribute disproportionately to loss reduction, whereas others are nearly redundant.

By Theophilus Amaefuna, Hitesh Vaidya, Anshuman Chhabra, Ankur Mali

llms efficiency

arXiv Machine Learning

Jul 28

Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

arXiv:2607. 22927v1 Announce Type: new Abstract: Weights and biases are normally optimized as separate parameter tensors, yet they do not represent separate functions when the input to an affine layer has nonzero mean.

By Zhang Gongyue, Sheng Yixuan, Liu donghan, Wang Zhiyong, Ren Weihong, Liu honghai

llms safety

arXiv Machine Learning

Aug 6

Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

arXiv:2608. 04382v1 Announce Type: new Abstract: Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization.

By Han Bao

safety

arXiv Machine Learning

Jun 9

Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy

arXiv:2606. 08779v1 Announce Type: new Abstract: Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses.

By Jiashun Liu, Runze Liu, Xu Wan, Jing Liang, Hongyao Tang, Ling Pan

llms reinforcement-learning

arXiv Machine Learning

Jul 31

Looped Transformers with Source-Centered State Evolution

arXiv:2607. 27656v1 Announce Type: new Abstract: Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count.

By Bum Jun Kim, Kohei Hayashi, Shunsuke Kamiya, Masanori Koyama, Yusuke Iwasawa, Yutaka Matsuo

llms benchmarks safety

arXiv Machine Learning

Jun 15

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

arXiv:2606. 13657v2 Announce Type: replace Abstract: On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student trajectories and dense teacher supervision.

By Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jun 25

Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.

By Alexander H\"agele, Alejandro Hern\'andez-Cano, Atli Kosson, Martin Jaggi

arXiv Machine Learning

Jul 20

Retraining Seeks Stable Signals

arXiv:2607. 15623v1 Announce Type: cross Abstract: Predictive models deployed at scale influence future data, a phenomenon called performativity.

By Moritz Hardt

llms

Related stories