The paper introduces the Retain-and-Repair Neural Operator (R$^2$NO), a method that adapts neural operators pretrained on numerical simulations to real-world data while preserving useful pretrained structure. R$^2$NO first fine‑tunes the pretrained operator on real data, freezes it to serve as a source prediction, and then employs a shared repair module that learns refinements via orthogonal Fourier projections and a spectral ensemble of ridge regressions. The approach allows refinement depth to vary across frequency magnitude, orientation, and channels, and consistently outperforms full fine‑tuning and iterative refinement on RealPDEBench systems across six backbones.
Neural operators increasingly benefit from pretraining on numerical simulations, yet adapting them for real-world prediction remains challenging. We introduce the Retain-and-Repair Neural Operator (R$...
arXiv:2606. 16602v1 Announce Type: new Abstract: Neural operator models trained on simulation data often lose accuracy when applied to experimental measurements due to the sim-to-real gap.
The paper investigates why latent neural surrogate solvers, which compress physical system dynamics into a lower‑dimensional space, often fail during long‑horizon autoregressive rollouts. It demonstrates that training the latent representation only for reconstruction leads to instability, and proposes a set of training interventions—Koopman operator learning, Hamming noise injection, and multi‑step rollout fine‑tuning—that align the latent space with long‑horizon forecasting. These interventions reduce long‑rollout error by about 40 % and achieve accuracy comparable to full‑resolution models while using far fewer floating‑point operations and GPU memory, enabling stable extrapolation in mesoscale crystal‑plasticity simulations of high‑cycle fatigue.
The paper introduces an in‑span adaptation technique for reduced‑order models, where the reduced subspace is continually updated using the model’s own predictions via an incremental singular‑value decomposition with a forgetting factor. This creates a trajectory‑informed spectral preconditioner that reweights and realigns the basis without changing the subspace, enabling the model to better absorb future out‑of‑span corrections. The authors demonstrate the method on a 3‑D spiral example and nonlinear PDEs such as viscous Burgers and Fisher–KPP, and relate the approach to in‑context learning in dynamical systems.
The paper introduces Foundation Preserving LoRA (FoLoRA), a forgetting‑aware optimization framework that balances adaptation to downstream tasks with preservation of pretraining capabilities. FoLoRA uses a first‑order preservation condition to define a forgetting penalty based on pretraining‑proxy activations and a task utility from downstream activations, scoring update directions via a generalized Rayleigh quotient. This spectral coordinate system enables gated Adam updates that reduce low‑utility, high‑penalty directions, and the method constructs pretraining proxy calibration data by sampling from the pretrained model. Experiments on math, code, and instruction‑following tasks demonstrate that FoLoRA achieves a stronger balance between target task performance and aggregate preservation of non‑target capabilities compared to baselines.
arXiv:2509. 11285v2 Announce Type: replace-cross Abstract: Class-Incremental Learning (CIL) in deep neural networks is conventionally framed as an iterative gradient-based optimization problem, incurring high computational cost, hyperparameter sensitivity, and risk of catastrophic forgetting.
arXiv:2606. 00132v1 Announce Type: cross Abstract: While finetuning effectively adapts foundation models to specialized downstream tasks, it can degrade nontarget capabilities acquired during pretraining.
SPARCL introduces a spectral partitioned analytic continual learning method that addresses forgetting in analytic class‑incremental learning. By decomposing the running autocorrelation into a high‑energy core and a residual complement, SPARCL freezes core components for old classes and updates only the residual block, ensuring closed‑form updates with an invariance guarantee. Experiments on CIFAR‑100, CUB‑200, ImageNet‑R, and ImageNet‑A with a frozen ViT‑B/16 protocol show that SPARCL narrows the performance gap between classical analytic learners and strong representation matchers while complementing sparse feature‑decorrelation approaches.
The paper proposes a drift‑adaptive method for Domain‑Incremental Learning that tracks representation drift for each past domain‑class pair. By measuring how much a concept’s current representation deviates from a fixed anchor, the method assigns stronger preservation to high‑drift pairs and down‑weights highly drifted domains during prototype construction. Experiments on DomainNet, Office‑Home, CORe50, and CDDB‑Hard show that this approach maintains a strong accuracy‑forgetting balance while using less training memory and the smallest storage compared to other methods.
arXiv:2606. 17567v1 Announce Type: new Abstract: While sequential residual fitting is the bedrock of standard boosting frameworks, it inherently breeds learner redundancy by repeatedly revisiting correlated error components.
RheOPT is a transformer-based neural operator that first pretrains on inexpensive Newtonian fluid simulations and then fine‑tunes on a small set of non‑Newtonian samples. By reusing geometry‑conditioned transport structures rather than the Newtonian constitutive response, RheOPT achieves accurate few‑shot predictions for complex fluids such as Giesekus, Oldroyd‑B, and Bingham in a contraction channel and for oscillatory wake flows behind a triangular obstacle. The approach demonstrates that Newtonian pretraining can match or surpass models trained from scratch with four times as many non‑Newtonian samples, especially when using reversible instance normalization.
arXiv:2505. 11766v4 Announce Type: replace Abstract: Neural Operators (NOs) are powerful architectures for learning mappings between function spaces.