arXiv Machine Learning

Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms

The paper presents a framework called auto‑differentiable filtering that jointly learns the state, dynamics, and filtering algorithm parameters in data assimilation. It uses a loss function that allows gradient‑based optimization from partial, noisy observations via auto‑differentiation. Experiments demonstrate the method on diverse dynamical systems, including aerospace, atmospheric, biological, and geophysical models, and the authors offer guidelines for tailoring the framework to specific observation models, accuracy needs, and computational budgets.

arXiv Machine Learning
Nov 6

A Survey of Nonlinear Filtering Methods for Data Assimilation and Recent Advances in Generative AI-Enabled Score Filters

The paper surveys nonlinear filtering methods for data assimilation, highlighting challenges in chaotic, high‑dimensional systems. It introduces the Ensemble Score Filter (EnSF), which encodes filtering densities via a score function and uses a training‑free Monte Carlo estimator to avoid Gaussian assumptions, particle collapse, and kernel scalability issues. The survey reviews EnSF’s mathematical foundations and fourteen extensions, noting that most retain the core estimator while two adapt it for planetary‑scale deployment.

By Cath Pham, Kadesha Reynolds
arXiv Statistics ML
Sep 17

A Continuous-Time Ensemble Kalman-Bucy Smoother for Causal Inference and Model Discovery

The paper presents an ensemble Kalman–Bucy smoother (EnKBS) for continuous‑time data assimilation of nonlinear dynamical systems, reconstructing conditional distributions from ensemble moments without needing tangent‑linear or adjoint models. It demonstrates that EnKBS achieves exact smoothing mean and covariance in the infinite‑ensemble limit for linear‑Gaussian systems and incorporates regularization techniques like covariance localization and inflation for high‑dimensional problems. The method is applied to Bayesian inference of causal relationships in a dyadic trigger‑feedback model and to an iterative learning algorithm that uncovers the structure and hidden parameters of a reduced‑order model of midlatitude atmospheric circulation, all with small ensembles under partial observations.

By Zhang Jiang (University of Wisconsin-Madison), Marios Andreou (University of Wisconsin-Madison), Sebastian Reich (University of Potsdam), Nan Chen (University of Wisconsin-Madison)
arXiv Statistics ML
Sep 3

The Ensemble Kalman Inversion Race

The paper compares different Ensemble Kalman methods for calibrating climate model parameters by minimizing the misfit between modeled and observed climate statistics. It conducts systematic numerical experiments on Lorenz-type models, including neural network parameterizations, to evaluate computational efficiency and accuracy of each method. The study examines how prior information and dimensionality affect the cost of these methods.

By Rebecca Gjini, Matthias Morzfeld, Oliver R. A. Dunbar, Tapio Schneider
arXiv Machine Learning
Aug 28

A Dynamic Likelihood Approach to Filtering for Advection-Diffusion Dynamics

A Bayesian data assimilation scheme called the Dynamic Likelihood Filter (DLF) is extended to handle advection-diffusion problems via a split-step formulation. The method propagates probabilities along characteristics using observations and their statistics, allowing the posterior estimate to inherit phase information. In advection-dominated scenarios, the DLF outperforms other assimilation approaches, especially when observations are sparse and have low uncertainty, though its computational cost scales cubically with the number of observations over time.

By Johannes Krotz, Juan M. Restrepo, Jorge Ramirez