Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression
Read the original on arXiv Machine Learning →The paper investigates continual linear regression in high-dimensional settings, deriving a closed-form expression for the generalization loss when using gradient descent with a fixed number of steps per task. It shows that the optimal fixed step size decays roughly as “ln T / T” for a single teacher and i.i.d. teachers, matching the performance of optimally tuned L2 regularization. The study also extends to generalized ridge regression with diagonal regularizers, identifies conditions for isotropic optimality, and reveals a regime where the step size stays finite as the number of tasks grows, highlighting a tradeoff between task variability and implicit regularization.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.