What Fourier analysis misses The post Sequential Fitting: A Different Perspective on the Spectral Bias of Neural Networks appeared first on Towards Data Science .
arXiv:2510. 19399v2 Announce Type: replace Abstract: Spectral bias, the tendency of neural networks to learn low-frequency features first, is a well-known issue with many training algorithms for physics-informed neural networks (PINNs).
The paper investigates when frequency decomposition aids Physics-Informed Neural Networks (PINNs) by introducing a dual‑branch, spectrally‑gated architecture (DBSG‑PINN) that separates low‑ and high‑frequency components. Experiments on five one‑dimensional PDE benchmarks show that frequency decomposition significantly reduces error—up to 59.2% on a multimodal wave problem—when the target solution is spectrally complex, but offers little improvement on smoother problems and can even worsen performance on a simple 1D wave benchmark. The adaptive gate’s effectiveness scales with the spectral richness of the solution, suggesting it exploits frequency structure rather than adding noise.
Physics-informed neural networks (PINNs) often fail to accurately resolve partial differential equations (PDEs) with high-frequency or multi-scale solutions, as well as strongly nonlinear problems. Two factors underlie this difficulty: spectral bias, the tendency of neural networks to underfit high-frequency features; and representation-coefficient coupling, the entanglement of representation learning and coefficient fitting within a single nonconvex optimization objective.
We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization. Extending Theorem 1 from Pesme & Flammarion (2023), we generalize the analysis to both deep diagonal linear networks and a broader class of two-layer diagonal linear networks (as defined in Definition 4.
The study compares two methods for computing spatial derivatives in physics‑informed neural networks (PINNs): automatic differentiation (AD) and Fourier spectral differentiation. Using identical neural architectures, training schedules, and data sampling, the authors evaluate both approaches on periodic PINNs for the Allen–Cahn, Korteweg–de Vries, and Kuramoto–Sivashinsky equations. Fourier spectral differentiation achieves significant speedups (2.90×–18.52×) and reduces GPU memory usage by 68.7%–94.1% while maintaining comparable solution accuracy.
arXiv:2608. 05892v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) often fail to accurately resolve partial differential equations (PDEs) with high-frequency or multi-scale solutions, as well as strongly nonlinear problems.
The paper introduces the Frequency Selective Neural Network (FSNN), a new foundation architecture for time‑series learning that embeds advanced signal‑processing mathematics into its neural topology. By using a fully differentiable Wiener‑like filter bank optimized with complex‑domain backpropagation, FSNN autonomously discovers and isolates the precise physical modes of a given task, thereby avoiding the spectral entanglement that plagues CNNs, RNNs, and Transformers. Extensive evaluations show that FSNN achieves state‑of‑the‑art predictive performance, attaining 77.0 % average accuracy on the 10 multivariate UEA datasets and leading all major metrics on the imbalanced PTB‑XL ECG benchmark, while converging directly on physically meaningful frequency bands such as the cardiac QRS complex.
arXiv:2606. 28486v1 Announce Type: cross Abstract: The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open problem.
arXiv:2606. 16575v1 Announce Type: new Abstract: Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors.
arXiv:2607. 12332v1 Announce Type: new Abstract: We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization.
arXiv:2602. 09530v2 Announce Type: replace-cross Abstract: We introduce AutoSpec, a neural network framework for discovering iterative spectral algorithms for large-scale numerical linear algebra and numerical optimization.
Let's discover how neural networks learn, step by step The post Backpropagation Explained for Beginners (Part 1): Building the Intuition appeared first on Towards Data Science .