arXiv AI
Jun 18
Surrogate Benchmarks for Model Merging Optimization
arXiv:2509. 02555v2 Announce Type: replace-cross Abstract: Model merging techniques aim to integrate the abilities of multiple models into a single model.
Multi-Objective Bayesian Optimization for Model Merging
arXiv:2608. 14264v1 Announce Type: cross Abstract: Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning.
Related stories
arXiv Machine Learning
1d ago
Mixture-Trained Merging for Unified Multi-Objective Models
Mixture-Trained Merging (MTM) is a method for creating unified language models that combine multiple objectives—such as mathematics, code, instruction following, and controllable thinking—into a single parameter set. Instead of sequentially post‑training on each objective, MTM trains each branch on a mixture of objectives, ensuring that the branches remain compatible in weight space and can be merged without degrading performance. The approach iteratively refines merge coefficients using low‑cost evaluations and multi‑objective Bayesian optimization, outperforming naive merging and preserving distinct behaviors across domains.
arXiv AI
Jul 13
LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms
arXiv:2607. 08791v1 Announce Type: cross Abstract: Designing effective multi-objective Bayesian optimization (MOBO) algorithms requires balancing many interdependent design choices whose optimal configuration is problem-dependent and typically demands deep expertise.
arXiv Machine Learning
Jun 10
mlr3mbo: Bayesian Optimization in R
arXiv:2603. 29730v2 Announce Type: replace-cross Abstract: We present mlr3mbo, a modular toolbox for Bayesian optimization in R.
arXiv Machine Learning
Sep 24
tidyHEBO: Robust General-Purpose Bayesian Optimization with Model-Consistent Warping and Pareto Search
tidyHEBO is a BoTorch-native Bayesian optimization tool that jointly applies Yeo-Johnson output warping to a Gaussian‑process surrogate, evaluates acquisition functions on the original objective scale, and conducts constrained cumulative Pareto search across multiple acquisition criteria. Using only default settings, it outperformed other methods on the Olympus benchmark and performed strongly on synthetic, Needle‑in‑a‑Haystack, and Bayesmark tasks, while adaptive batching offered a trade‑off between parallelization and optimization quality. These results position tidyHEBO as a robust, reproducible optimizer suitable for diverse practical problems, including scientific applications and hyperparameter tuning.
arXiv AI
Sep 15
Converge Then Diversify: Decoupling Convergence and Diversity in Multi-Objective Bayesian Optimisation
arXiv:2609.13396v1 Announce Type: new Abstract: Multi-objective Bayesian optimisation (MOBO) is a sample-efficient approach for optimising expensive black-box functions with multiple objectives. In M...
arXiv Machine Learning
Aug 11
A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization
arXiv:2608. 08156v1 Announce Type: new Abstract: In evolutionary algorithms powered by language models, the LLM acts as a single operator that simultaneously updates structural components (like control flow) and continuous parameters.
arXiv AI
Aug 13
High-dimensional Multi-objective Bayesian Optimization with Learned Variable Interactions
arXiv:2608. 11713v1 Announce Type: cross Abstract: Multi-objective Bayesian optimization (MOBO) is effective in identifying the Pareto fronts for expensive black-box problems.
arXiv AI
Jun 30
Model Merging to Evolution: Parameter Space Exploration for Expert Models
arXiv:2606. 28373v1 Announce Type: cross Abstract: Model merging integrates the capabilities of multiple expert models to create strong models for multiple tasks without additional training, thereby reducing computational resource requirements.
arXiv AI
Jun 2
FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
arXiv:2605. 25246v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines.
arXiv AI
Aug 14
CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
arXiv AI
Jun 3
When Model Merging Breaks Routing: Training-Free Calibration for MoE
arXiv:2606. 03391v1 Announce Type: cross Abstract: Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining.