arXiv:2610. 11475v1 Announce Type: new Abstract: Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining.
arXiv:2606. 21253v2 Announce Type: replace Abstract: Continual learning that is gradient-free, local, online, and append-only is attractive for edge and streaming deployment, but its value is usually argued informally.
arXiv:2609.36945v1 Announce Type: new Abstract: We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version...
The paper investigates how momentum methods affect large‑batch training in a one‑pass setting using power‑law kernel regression. It derives critical learning rates for SGD, Polyak, and Nesterov, and shows how these rates depend on batch size, momentum, and model capacity. The authors provide scaling laws for risk dynamics, a three‑regime batch‑size phase diagram, and demonstrate that Polyak increases the critical batch size while Nesterov improves data efficiency in the large‑batch regime.
arXiv:2605. 24316v2 Announce Type: replace Abstract: Scaling laws provide compact descriptions of how prediction error varies with compute, model size, and data, but existing theory mainly treats single-sample SGD or full data reuse, leaving the role of mini-batching unclear.
The paper presents a theoretical study of Adam in non‑stationary stochastic optimization, distinguishing two regimes: Euclidean tracking under adaptive strong monotonicity and high‑probability projected stationarity for general smooth objectives. It derives finite‑time bounds that decompose into initialization, objective drift, first‑moment tracking error (β₁), and preconditioner perturbation (β₂), and characterizes burn‑in times for constant and step‑decay schedules. The analysis reveals a noise–drift tradeoff, showing that in noise‑dominated settings Adam’s adaptive mechanisms can improve guarantees, while in drift‑dominated settings they may worsen tracking, potentially making vanilla SGD preferable.
arXiv:2608. 07922v1 Announce Type: new Abstract: Adaptive learning needs both a state that preserves what observations imply and opportunities to act on that state.
The paper demonstrates that diffusion models encode memorization in the geometry of their learned energy landscape before it becomes visible in generated samples. By analyzing score divergence and basin volume, the authors show that localized basins form around training images early in training, and that cyclic denoising can recover these images even when one‑shot generation shows no copies. Experiments on Gaussian mixtures, CelebA, CIFAR‑10, and Stable Diffusion v1.4 confirm that basin geometry reveals memorization across various settings.
arXiv:2602. 02431v2 Announce Type: replace-cross Abstract: It is folklore that reusing training data more than once can improve the statistical efficiency of gradient-based learning.
The paper identifies a problem called Gradient-Update Mismatch (GUM), where optimizers can alter conflict-free gradient directions produced by gradient surgery, leading to conflicts between physics residual and boundary condition losses in Physics-Informed Neural Networks (PINNs). To address this, the authors propose Gradient-Update Alignment (GUA), which projects the optimizer’s update onto the conflict-free cone and adjusts internal optimizer state accordingly. Experiments show GUM is common across many optimizers, and GUA consistently eliminates conflicts and significantly reduces error in PINN training.
arXiv:2609.38880v1 Announce Type: new Abstract: We study the last iterate of standard tabular temporal-difference (TD) learning from a single trajectory of a finite Markov reward process. For discoun...
Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$, and if it fades too fast the dependence cannot be learned from finite data. This fade is captured by an envelope $f(\ell)$.