arXiv Machine Learning
Jul 9
Information Allocation Dynamics in Neural Network Optimization
arXiv:2607. 07156v1 Announce Type: new Abstract: Different optimizers have different update biases, but these biases are usually implicit.
The Anatomy of Implicit Bias: Information Allocation in Neural Network Training
arXiv:2607. 07156v2 Announce Type: replace Abstract: Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry.
Related stories
arXiv Machine Learning
Aug 10
Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learning
arXiv:2608. 07157v1 Announce Type: new Abstract: Sub-model federated learning lets resource-constrained clients train width-reduced versions of a global model, but existing methods allocate capacity by device resources alone.
arXiv AI
5d ago
Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller
arXiv:2610.18294v1 Announce Type: cross Abstract: Releasing a neural network checkpoint can leak information about its training data, and existing reconstruction results rely on gradient-descent trai...
arXiv Machine Learning
Aug 13
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
arXiv:2608. 11690v1 Announce Type: new Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.
arXiv Machine Learning
1d ago
Error Correction in Trained TTT Layers
arXiv:2610.16268v2 Announce Type: replace Abstract: Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself...
arXiv Machine Learning
1d ago
TALLOP: Joint Optimization of Local Training and Communication Compression for Time-Constrained Federated Learning
TALLOP is a method for federated learning that jointly optimizes local training workload and communication compression within a fixed per‑round time budget. It predicts achievable training loss from early mini‑batch loss trajectories (LENS) and estimates compression‑induced degradation using a compression‑error model (CLEN). By combining these predictions, TALLOP selects a training‑communication allocation that minimizes post‑compression loss, achieving competitive or improved time‑to‑accuracy in heterogeneous federated learning environments.
arXiv Machine Learning
Aug 3
Optimal Resource Allocation for ML Model Training and Deployment under Concept Drift
arXiv:2512. 12816v2 Announce Type: replace Abstract: We study how to allocate resources for training and deployment of machine learning (ML) models under concept drift and limited budgets.
arXiv AI
Oct 2
FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection
FedLore introduces a communication- and memory-efficient federated learning framework that shares a low-rank optimization basis across clients each round, mitigating subspace fragmentation and enabling exact low-rank aggregation. By refreshing this shared basis across rounds, FedLore allows model updates to exceed the per-round rank budget while maintaining a provable $O(T^{-1/2})$ stationarity bound under standard assumptions. Experiments on vision and language tasks, including federated pre‑training, demonstrate that FedLore outperforms low‑rank adapter baselines and matches or surpasses full‑parameter training while reducing communication and optimizer‑state memory.
arXiv Machine Learning
Jun 2
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
arXiv:2606. 00340v1 Announce Type: new Abstract: We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions.
arXiv Machine Learning
Sep 10
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
arXiv:2609.07755v1 Announce Type: new Abstract: Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training d...
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv Machine Learning
Jul 31
Looped Transformers with Source-Centered State Evolution
arXiv:2607. 27656v1 Announce Type: new Abstract: Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count.