The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2506. 01052v3 Announce Type: replace Abstract: We investigate the finite-time convergence properties of Temporal Difference (TD) learning with linear function approximation, a cornerstone of reinforcement learning.
arXiv:2606. 05967v1 Announce Type: cross Abstract: In this paper, we study the finite-time behavior of the TD(0) temporal-difference method with linear function approximation (LFA).
The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.
In this paper, we study the finite-time behavior of the TD(0) temporal-difference method with linear function approximation (LFA). We consider on-policy independent and identically distributed (i.
arXiv:2609. 29961v1 Announce Type: new Abstract: Many iterative algorithms rely on bootstrapping.
The paper introduces a contraction framework for stochastic operators that incorporates bootstrapping, where a variable is updated using a frozen copy as a target that is refreshed every $K$ steps. By modeling the sampled update as a stochastic operator, the authors derive a finite‑time bound for i.i.d. samples that applies to any target‑update period and does not require gradient structure or uniformly bounded sampling error. The framework shows that the iterates converge geometrically in root mean square to a ball around the fixed point, with the error floor scaling with the step size, and it generalizes existing deterministic and stochastic‑gradient bounds.