The paper surveys continuous‑time (CT) machine learning, a framework for modeling temporal dynamics as continuous processes, especially useful when data are sampled irregularly or over long horizons. It introduces a unified taxonomy that groups major CT methods by their underlying mathematical formulations and shows how different architectural choices—such as vector‑field parameterization, stochasticity, memory mechanisms, and discretization—relate these families. The survey compares training algorithms, optimization strategies, failure modes, computational complexity, and benchmarks, reviews supporting software ecosystems, and outlines open challenges and future research directions.
By Waleed Razzaq, Yun-Sheng Zhao, Yun-Bo Zhao
arXiv:2606. 03831v1 Announce Type: new Abstract: This paper investigates non-stationary online learning using the metric of interval regret, which requires an online algorithm to perform well over every time interval.
By Yan-Feng Xie, Shuche Wang, Peng Zhao, Zhi-Hua Zhou
arXiv:2405. 04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.
By Yijiang Pang, Shuyang Yu, Bao Hoang, Jiayu Zhou
Continuous-time (CT) machine learning has emerged as a principled framework for modeling temporal dynamics as a continuous process, particularly when observations are sampled at arbitrary time points...
arXiv:2606. 01827v1 Announce Type: cross Abstract: Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models.
By Dimitris Oikonomou, Nicolas Loizou
ExpTest is an autonomous learning‑rate controller that uses the training loss curve as an online signal to perform sequential statistical tests on theoretically motivated windows, detecting convergent behavior and triggering learning‑rate reductions. It combines a covariance‑based initial learning‑rate estimate, curvature‑motivated window sizing, and a two‑phase test‑driven decay, relying on the approximately exponential decay predicted under linearized network dynamics. Experiments on regression, classification, forecasting, and natural‑language tasks across various architectures show that ExpTest achieves competitive performance compared to hand‑tuned SGD baselines and recent learning‑rate‑free methods, without requiring manual initial learning‑rate selection or predefined scheduling.
By Zan Chaudhry, Naoko Mizuno
HUANet is a deep neural network architecture that unrolls the Alternating Direction Method of Multipliers (ADMM) into a trainable model for accelerating parametric constrained convex optimization. It embeds a hard‑constrained neural network in each ADMM iteration, using a differentiable correction stage to enforce affine equalities of the primal subproblem. The method also incorporates first‑order optimality conditions into a self‑supervised training loss, and numerical experiments on benchmark problems and a control application demonstrate its effectiveness in speeding up constrained convex optimization.
By Trinh Tran, Binh Nguyen, Truong X. Nghiem
arXiv:2607. 09167v1 Announce Type: new Abstract: Schedule-Free methods have attracted growing interest for alleviating the burden of designing and tuning a learning rate scheduler, while matching and sometimes even outperforming optimizers with tuned schedulers.
By Jiseok Chae, Donghwan Kim
arXiv:2608. 25551v1 Announce Type: new Abstract: Stochastic gradient descent (SGD) is typically analyzed at a deterministic horizon chosen before the algorithm is run, even though practical stopping decisions are made adaptively by inspecting the evolving trajectory.
By Liviu Aolaritei, Lucas L\'evy, Francis Bach, Michael I. Jordan
arXiv:2406.04261v2 Announce Type: replace
Abstract: Simulators are vital in science and engineering, as they faithfully model the influence of design parameters on real-world observations. Optimizing...
By Tim Bakker, Fabio Valerio Massoli, Thomas Hehn, Tribhuvanesh Orekondy, Arash Behboodi
arXiv:2605.18618v3 Announce Type: replace-cross
Abstract: Constrained Machine Learning (CML) enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain know...
By Adam Bos\'ak, Andrii Kliachkin, Gilles Bareilles, Allen Gehret, Allahkaram Shafiei, Jana Lep\v{s}ov\'a, Jakub Mare\v{c}ek
arXiv:2412. 19444v2 Announce Type: replace Abstract: Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process.
By Yuanzhe Tao, Yifeng Liu, Huizhuo Yuan, Xun Zhou, Yuan Cao, Quanquan Gu