arXiv Machine Learning
Jul 9
Information Allocation Dynamics in Neural Network Optimization
arXiv:2607. 07156v1 Announce Type: new Abstract: Different optimizers have different update biases, but these biases are usually implicit.
The Anatomy of Implicit Bias: Information Allocation in Neural Network Training
arXiv:2607. 07156v2 Announce Type: replace Abstract: Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry.
Related stories
arXiv Machine Learning
Aug 10
Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning
arXiv:2608. 07157v1 Announce Type: new Abstract: Sub-model federated learning lets resource-constrained clients train width-reduced versions of a global model, but existing methods allocate capacity by device resources alone.
arXiv Machine Learning
6d ago
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
arXiv:2608. 11690v1 Announce Type: new Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.
arXiv Machine Learning
Aug 3
Optimal Resource Allocation for ML Model Training and Deployment under Concept Drift
arXiv:2512. 12816v2 Announce Type: replace Abstract: We study how to allocate resources for training and deployment of machine learning (ML) models under concept drift and limited budgets.
arXiv Machine Learning
Aug 4
The Fourth Quadrant: A Stylized View of Benign Misfitting
arXiv:2608. 01032v1 Announce Type: new Abstract: Training error is what we can observe on a training set; test error is the quantity we actually care about.
arXiv Machine Learning
Jun 2
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
arXiv:2606. 00340v1 Announce Type: new Abstract: We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv Machine Learning
Jul 31
Looped Transformers with Source-Centered State Evolution
arXiv:2607. 27656v1 Announce Type: new Abstract: Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count.
arXiv AI
Jun 4
An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
arXiv:2606. 04752v1 Announce Type: cross Abstract: Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_{\text{model}}$-dimensional vector per time step.
arXiv Machine Learning
Aug 6
Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics
arXiv:2608. 04382v1 Announce Type: new Abstract: Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization.
arXiv Machine Learning
Jun 5
Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning
arXiv:2605. 27991v2 Announce Type: replace-cross Abstract: Gradient-flow optimization is usually viewed as an algorithmic procedure for minimizing empirical loss, with training duration selected by validation or heuristic early-stopping rules.
arXiv AI
Jun 9
HASA: Subnet Allocation for Compute-Constrained Model-Heterogeneous Federated Learning
arXiv:2606. 07621v1 Announce Type: cross Abstract: Edge services increasingly use federated learning to personalize on-device models while keeping sensitive data local.