arXiv Machine Learning
Aug 3
Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization
arXiv:2511. 02570v3 Announce Type: replace Abstract: Bayesian optimization (BO) is a widely used approach to hyperparameter optimization (HPO).
Provably Reduced Sample Cost in Prior-Guided Hyperparameter Optimization
arXiv:2606. 04866v1 Announce Type: new Abstract: Large-scale hyperparameter optimization (HPO) in automated machine learning (AutoML) consumes substantial computational resources, raising growing concerns about scalability and energy efficiency.
Related stories
arXiv AI
Jun 2
c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
arXiv:2211. 14411v5 Announce Type: replace-cross Abstract: Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as on memory usage or latency, on top of the performance requirement.
arXiv Machine Learning
Sep 11
Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates
arXiv:2604. 13130v2 Announce Type: replace Abstract: We study learning to learn through the lens of hyperparameter tuning.
arXiv AI
Aug 20
Automated Computational Energy Minimization of ML Algorithms using Constrained Bayesian Optimization
The paper presents a method that uses Constrained Bayesian Optimization (CBO) to minimize the energy consumption of machine learning models while ensuring their generalization performance stays above a specified threshold. By treating energy usage as the primary objective and performance as a constraint, the authors demonstrate that CBO can reduce training energy costs on both regression and classification tasks without sacrificing predictive accuracy.
arXiv AI
Jun 3
q0: Primitives for Hyper-Epoch Pretraining
arXiv:2606. 03938v1 Announce Type: cross Abstract: Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text.
arXiv Machine Learning
Sep 3
HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC
HyperMC is a multi‑fidelity hyperparameter tuning framework for stochastic gradient Markov chain Monte Carlo (SGMCMC) that combines Hyperband-style resource allocation with kernel Stein discrepancy (KSD) evaluation. It uses successive‑halving brackets to explore a continuous hyperparameter space while progressively refining promising configurations within a fixed computational budget. Robust HyperMC further introduces global grid initialization and elite‑guided local refinement to reduce sensitivity to random candidate generation and noisy evaluations, and theoretical analysis shows that the successive‑halving component selects a near‑optimal configuration with high probability under suitable conditions.
arXiv AI
Sep 2
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
The paper introduces Power‑Law Entropy Search (PLES), a computational‑cost‑aware acquisition function that uses multi‑fidelity Bayesian optimization to efficiently estimate optimal hyperparameter scaling laws for large language model training. PLES focuses on reducing the overall uncertainty of scaling law estimates rather than optimizing a single objective, selecting configurations that maximize uncertainty reduction per unit computational cost. Experiments on synthetic benchmarks, surrogate models, and real LLM pre‑training runs show that PLES converges to accurate scaling laws using less than one‑tenth of the computational budget required by conventional grid search and other baselines.
Hugging Face Trending Papers
Jul 15
Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems
The rapid deployment of machine learning systems across cloud, edge, and enterprise environments has brought model optimization to the forefront of systems-engineering. Despite a rich literature spanning quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference-time optimization, practitioners are often left navigating these techniques through heuristics rather than principled methodology.
arXiv AI
Jul 21
Reducing Per-Sample Harm in Stochastic Optimization
arXiv:2607. 16261v1 Announce Type: cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.
arXiv AI
Aug 21
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
arXiv AI
Jun 3
Efficient Hyperparameter Optimization for LLM Reinforcement Learning
arXiv:2606. 03073v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language models (LLMs) is highly sensitive to hyperparameter configurations, making hyperparameter optimization (HPO) essential yet computationally expensive.
arXiv Machine Learning
Jul 16
Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems
arXiv:2607. 13735v1 Announce Type: new Abstract: The rapid deployment of machine learning systems across cloud, edge, and enterprise environments has brought model optimization to the forefront of systems-engineering.