arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
By Haeyong Kang, Chang D. Yoo
arXiv:2607. 05609v1 Announce Type: cross Abstract: The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting.
By Giulia Lanzillotta, Mandana Samiei, Doina Precup, Razvan Pascanu, Claire Vernade
arXiv:2504. 13822v3 Announce Type: replace-cross Abstract: The emergence of large pre-trained networks has revolutionized the AI field, unlocking new possibilities and achieving unprecedented performance.
By Eric Nuertey Coleman, Luigi Quarantiello, Ziyue Liu, Qinwen Yang, Samrat Mukherjee, Julio Hurtado, Vincenzo Lomonaco
arXiv:2604. 15613v4 Announce Type: replace-cross Abstract: We present Green-ELM, a non-iterative neural architecture that replaces gradient-based optimization of the output layer with a closed-form analytic solution over a fixed, high-dimensional random feature representation.
By Wladimir Silva
SPARCL introduces a spectral partitioned analytic continual learning method that addresses forgetting in analytic class‑incremental learning. By decomposing the running autocorrelation into a high‑energy core and a residual complement, SPARCL freezes core components for old classes and updates only the residual block, ensuring closed‑form updates with an invariance guarantee. Experiments on CIFAR‑100, CUB‑200, ImageNet‑R, and ImageNet‑A with a frozen ViT‑B/16 protocol show that SPARCL narrows the performance gap between classical analytic learners and strong representation matchers while complementing sparse feature‑decorrelation approaches.
By James Hartley, Zeropy Surio, Daniel Whitmore, Hannah Clarke, Thomas Reed
ExpTest is an autonomous learning‑rate controller that uses the training loss curve as an online signal to perform sequential statistical tests on theoretically motivated windows, detecting convergent behavior and triggering learning‑rate reductions. It combines a covariance‑based initial learning‑rate estimate, curvature‑motivated window sizing, and a two‑phase test‑driven decay, relying on the approximately exponential decay predicted under linearized network dynamics. Experiments on regression, classification, forecasting, and natural‑language tasks across various architectures show that ExpTest achieves competitive performance compared to hand‑tuned SGD baselines and recent learning‑rate‑free methods, without requiring manual initial learning‑rate selection or predefined scheduling.
By Zan Chaudhry, Naoko Mizuno