The paper argues against using the Gaussian (squared exponential/RBF) kernel as a default in Gaussian process regression, citing its brittleness. It shows that the kernel leads to unrealistically small conditional variances, causing overconfidence in predictive uncertainty, and that this small variance induces numerical ill‑conditioning, necessitating tricks like nugget terms that alter the model. The authors attribute these issues to the kernel’s analytic, highly smooth nature and suggest that analytic stationary kernels in general should be avoided.
The monograph explores the relationships between Gaussian processes and reproducing kernel Hilbert spaces (RKHS), two widely used approaches that rely on positive definite kernels. It examines how these frameworks connect and are equivalent across key topics such as regression, interpolation, numerical integration, distributional discrepancies, statistical dependence, and Gaussian process sample path properties. By establishing a unifying perspective based on the equivalence between the Gaussian Hilbert space and the RKHS, the work aims to bridge methods developed independently by the machine learning, statistics, and numerical analysis communities.
By Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, Bharath K. Sriperumbudur
arXiv:2610.10080v1 Announce Type: cross
Abstract: Gaussian processes (GPs) are widely used across machine learning, spatial statistics, time-series analysis, optimization, Bayesian statistics, and sc...
By Ameer Qaqish, Didong Li
arXiv:2512. 06143v2 Announce Type: replace Abstract: Despite a large corpus of recent work on scaling up Gaussian processes, a stubborn trade-off between computational speed, prediction and uncertainty quantification accuracy, and customizability persists.
By Marcus M. Noack, Mark D. Risser, Hengrui Luo, Vardaan Tekriwal, Ronald J. Pandolfi
arXiv:2510. 25599v2 Announce Type: replace Abstract: Regression tasks, notably in safety-critical domains, require reliable uncertainty quantification, yet the literature remains largely classification-focused.
By Christopher B\"ulte, Yusuf Sale, Gitta Kutyniok, Eyke H\"ullermeier
arXiv:1812.09632v3 Announce Type: replace
Abstract: We propose a data-driven approach to quantify the uncertainty of models constructed by kernel methods. Our approach minimizes the needed distributi...
By Bal\'azs Csan\'ad Cs\'aji, Kriszti\'an Bal\'azs Kis
arXiv:2603. 16481v3 Announce Type: replace Abstract: Non-conservative uncertainty bounds are essential for making reliable predictions about latent functions from noisy data, and thus, a key enabler for safe learning-based control.
By Amon Lahr, Anna Scampicchio, Johannes K\"ohler, Melanie N. Zeilinger
arXiv:2610.03201v1 Announce Type: cross
Abstract: Gaussian Processes (GPs) are a powerful tool for modelling and quantifying uncertainty in functional relationships. However, they require practitione...
By Callum Lau, Jeremias Knoblauch, Louis Sharrock
arXiv:2605. 10285v2 Announce Type: replace-cross Abstract: We present a theoretically grounded Gaussian process framework that leverages neural feature maps to construct expressive kernels.
By Anthony Stephenson
The paper introduces SCORE, a scalable framework that learns an expressive covariance approximation for high‑dimensional Gaussian targets by combining scoring rule training with spectral‑space modeling. It decomposes the learning task into marginal distribution estimation and structured correlation matrix learning, achieving dense dependencies with linear storage and ≠(d log d) computational cost. The method employs a closed‑form Gaussian kernel score that remains valid for degenerate covariances, provides bounded gradients, and is invariant under unitary transforms, with theoretical guarantees on marginal recovery and correlation projection, and demonstrates improved performance on time‑series forecasting, monocular depth estimation, and spatial weather prediction.
By Christopher B\"ulte, Emil Partow, Astha Gupta, Pascal Esser, Gitta Kutyniok
The paper introduces HACK GPs, a method that treats kernel selection for Gaussian Processes as an online learning problem with expert advice. Each candidate kernel is viewed as a GP expert, and a distribution over these experts is updated online using AdaHedge based on a loss that reflects both function fit and task alignment. Two variants—Mixture of Gaussians and categorical sampling—are presented, with theoretical guarantees that the weight concentrates on the best kernel under a loss‑gap condition, and empirical results show robust performance across Bayesian optimization, level set estimation, and Bayesian active learning compared to standard kernels and simple ensembles.
By Kavin Aravindan, Mani Tej Sriram, Gautam Dasarathy, Tejas Bodas
arXiv:2608. 13793v1 Announce Type: cross Abstract: Machine learning (ML) has become an indispensable part of modern engineering design workflows.
By Tyler R. Johnson, Kian Ben-Jacob, Christopher P. Muller, Ramin Bostanabad