arXiv:2602.06797v3 Announce Type: replace-cross
Abstract: We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training d...
By Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu
arXiv:2610.06894v1 Announce Type: cross
Abstract: A central question in the prediction of stochastic processes is the extent to which past information can improve the probability of correctly predict...
By Jiahao Jiang
arXiv:2605. 05113v2 Announce Type: replace Abstract: We study signal propagation in linear recurrent models at finite width.
By Mariia Seleznova
The paper shows that a memory state that reproduces every current prediction can still fail to learn from subsequent observations. It characterizes when a fixed‑dimensional continuous recurrent state exists for exact cumulative learning and attention, proving that for smooth convex residual losses the state exists only when the loss is polynomial. The work further quantifies the dimensionality required for various loss functions, demonstrates unbounded requirements for nonpolynomial losses, and analyzes specific cases such as ridge regression and kernel attention.
By Robert Jenkinson Alvarez
arXiv:2610.07623v1 Announce Type: cross
Abstract: Probability forecasts are calibrated when predicted probabilities match empirical outcome frequencies: among events assigned a probability $p$, we'd...
By Eric Dai, Maxwell Fishelson
Fast Weight Attention for Continual Learning introduces recurrent fast‑weight memories and selective state‑space models that compress expanding context into a fixed‑size recurrent state, enabling an online learning rule for state transitions. The paper derives normalized first‑order updates for squared‑error regression and negative inner‑product objectives, presenting several variants (Falcon‑1, Falcon‑2, Falcon‑3 and their inner‑product counterparts) with recurrent, masked‑parallel, and chunk‑parallel implementations. These methods demonstrate competitive performance in language modeling and improved length extrapolation on variable‑digit addition tasks.
By Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, Andrew Chi-Chih Yao