LoRA-RC introduces a low‑rank adaptation scheme for reservoir computing that updates the recurrent matrix using streaming prediction errors while keeping the base reservoir and adaptation bases fixed offline. The method projects a small core matrix onto a spectral‑norm ball and applies low‑pass filtering at each step, ensuring every recurrent matrix stays within a certified contraction set. Experiments on a Lorenz system with abrupt parameter drift show that LoRA‑RC reduces post‑drift prediction error by 56% compared to a fixed RC and 51% compared to readout‑only adaptation, and that removing the projection increases error by more than a factor of 40.
arXiv:2608. 04028v1 Announce Type: cross Abstract: Echo-state networks enable efficient temporal learning by fixing the recurrent dynamics and training only a linear readout.
The paper introduces a data‑adaptive nonlinear vector autoregression (NVAR) model that replaces fixed polynomial or random feature maps with a shallow, trainable multilayer perceptron (MLP). By jointly training the MLP and a linear readout via gradient‑based optimization, the model learns data‑driven nonlinearities while maintaining a simple readout structure, improving scalability in high‑dimensional settings. Experiments on several chaotic systems, both noise‑free and synthetically noisy, show that this adaptive NVAR outperforms standard NVAR, a leaky echo state network (ESN), and a hybrid ESN in predictive accuracy, demonstrating robust forecasting under noisy conditions.
Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization.
arXiv:2509. 24122v3 Announce Type: replace Abstract: At the heart of time-series forecasting (TSF) lies a fundamental challenge: how can models efficiently and effectively capture long-range temporal dependencies across ever-growing sequences?
arXiv:2606. 12141v1 Announce Type: new Abstract: Accurate forecasting of sea surface temperature (SST) in regional seas such as the East Sea is crucial for monitoring marine ecosystems, assessing climate risks, managing fisheries, and conducting naval operations.
arXiv:2608. 04593v1 Announce Type: cross Abstract: Echo State Networks (ESNs) offer an efficient framework for temporal prediction, but their randomly initialized reservoirs are often over-parameterized and dynamically redundant.
arXiv:2508. 21172v3 Announce Type: replace-cross Abstract: Echo State Networks (ESNs) are a particular type of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular for their fast and efficient learning.
arXiv:2607. 17909v1 Announce Type: cross Abstract: The edge-of-chaos heuristic has long served as a guiding principle for designing reservoir computers, yet its relevance to machine performance remains elusive.
Normalized Low-Rank Adaptation (NoRA) is a lightweight enhancement to the widely used LoRA technique that normalizes the down‑projection matrices during training. By doing so, NoRA stabilizes early optimization dynamics, accelerates convergence, and improves performance across pretraining, supervised fine‑tuning, and reinforcement learning. The method adds no extra trainable parameters or inference‑time cost, making it broadly applicable.
LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.
Echo State Networks (ESNs) offer an efficient framework for temporal prediction, but their randomly initialized reservoirs are often over-parameterized and dynamically redundant. Existing pruning methods largely rely on static connectivity or activation statistics, which may overlook neurons that shape input-driven state transitions.
TaRA: Training-Aware Low-Rank Adaptation Initialization proposes a new way to initialize LoRA by aligning the gradients of low‑rank factors with those of the full‑rank weight matrix. This approach directly incorporates training dynamics, improving gradient fidelity at the start of fine‑tuning while adding negligible computational cost. Experiments on a variety of challenging fine‑tuning tasks show that TaRA consistently outperforms existing state‑of‑the‑art initialization methods, offering a simple, robust, and scalable solution for effective LoRA initialization.