The paper introduces TGL-NSGA-II, a low‑fidelity framework that uses a pretrained teacher to stratify samples by difficulty and class, then applies a short knowledge‑distillation step (KD‑Lite) before scoring candidates on a stratified evaluation set. The teacher‑guided scores are fused with a Gaussian‑process surrogate to select candidates for full evaluation, and the method is evaluated on keyword spotting and bird‑call classification tasks. Results show high Kendall‑τ values (0.74 and 0.62), a 41% reduction in proxy‑score variance, and improved hypervolume and false‑positive rates compared to full NSGA‑II, while running 2.2× faster under a constrained evaluation budget.
arXiv:2510. 06048v4 Announce Type: replace Abstract: Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks.
arXiv:2606. 00862v1 Announce Type: cross Abstract: Surrogate-assisted evolutionary algorithms (SAEAs) have been widely used for expensive black-box optimization problems.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv:2609.25745v1 Announce Type: new Abstract: RandOpt samples weight-perturbed language models and ensembles top-ranked candidates through plurality voting, but its global perturbation scale ignore...
tidyHEBO is a BoTorch-native Bayesian optimization tool that jointly applies Yeo-Johnson output warping to a Gaussian‑process surrogate, evaluates acquisition functions on the original objective scale, and conducts constrained cumulative Pareto search across multiple acquisition criteria. Using only default settings, it outperformed other methods on the Olympus benchmark and performed strongly on synthetic, Needle‑in‑a‑Haystack, and Bayesmark tasks, while adaptive batching offered a trade‑off between parallelization and optimization quality. These results position tidyHEBO as a robust, reproducible optimizer suitable for diverse practical problems, including scientific applications and hyperparameter tuning.
arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.
ESPO (Error-Structured Prompt Optimization) addresses prompt bloat in evolutionary prompt optimizers by splitting the optimization process into Diagnose, Propose, and Select phases. It clusters training errors into structural patterns, generates diverse candidate prompts through four complementary strategies, and applies bootstrap stability selection. Across seven NLP benchmarks, ESPO improves average accuracy by +3.76 pp over GEPA, produces prompts 47 % shorter, and achieves higher accuracy on four additional student models, with the largest gain on Qwen3 GSM8K.
arXiv:2606. 25488v1 Announce Type: new Abstract: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments.
ESPO (Error-Structured Prompt Optimization) addresses prompt bloat in evolutionary prompt optimizers by separating optimization into Diagnose, Propose, and Select phases. It clusters training errors, generates diverse candidates, and applies bootstrap stability selection, achieving a 3.76‑point accuracy gain over GEPA on seven NLP benchmarks while producing 47% shorter prompts. Cross‑model tests on four additional student models confirm ESPO’s superior average accuracy, notably improving Qwen3 GSM8K from 15.00% to 91.40%.
SkillLift introduces a method for efficiently evolving reusable procedural prompts (skills) in large language model agents by learning a dense rubric that aligns with sparse oracle evaluations. Instead of directly revising skill text based on costly full agent rollouts, the approach decouples skill search from oracle cost through a bilevel optimization framework: an inner loop uses a frozen rubric as a cheap surrogate to guide skill updates, while an outer loop periodically realigns the rubric using a small number of oracle rollouts via rank correlation. Experiments on complex agent task benchmarks demonstrate that SkillLift outperforms existing auto-skill methods while reducing token cost by 40–70% compared to frontier-evolving approaches.
arXiv:2608. 02391v1 Announce Type: cross Abstract: Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive.
Evolutionary Ensemble Search (EES) is a framework that builds machine‑learning procedures through expert‑guided program evolution. A specialized council interprets task evidence and experimental results to generate structured search directions, which an orchestrator assigns to execution specialists and an evolutionary engine. The engine selects parents, diagnoses errors, and creates descendants via code mutation, pipeline edits, and crossover, with each child evaluated on its own validation evidence. Population archives preserve useful alternatives, and compatible predictions compete in a validation‑gated ensemble stage. Search adapts through parent‑relative operator credit, session memory, and lessons retrieved across runs. The system achieved medal‑threshold artifacts on 19 of 22 tasks (86.36 %) with 11 gold, five silver, and three bronze outcomes across diverse modalities.