The paper introduces FARO, a function‑space adaptive optimizer that treats each parameter block as a decision unit, assigning geometry based on counterfactual return and reallocating budgets via cross‑block equal risk contribution (ERC). On a GPT‑style model trained on FineWeb, FARO‑Adaptive matches the best hand‑designed optimizer (NorMuonAndAdam) while FARO‑ERC further reduces clean‑data loss by an average of 0.0136 nats across six seeds, achieving a robust improvement without extra time or memory. The method works by trimming the riskiest blocks and remains effective even under corrupted data streams.
arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...
arXiv:2606. 21641v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that "warm-start" search from prior knowledge, proposing strong configurations in very few evaluations.
arXiv:2607. 27787v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability.
arXiv:2603. 00910v2 Announce Type: replace-cross Abstract: Layer-wise capacity in large language models is highly non-uniform: some layers contribute disproportionately to loss reduction, whereas others are nearly redundant.
Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient.
The paper introduces Adaptive Multi Mode (AMM), a budget‑aware router that selects among several optimization engines—quasi‑Newton, trust‑region, coordinate, multistart, and evolution‑strategy—based on online probes of the problem landscape. Each probe evaluates features such as curvature, plateaus, separability, kinks, basins, and fractal structure, and is assigned a validity range and cost relative to the budget. In extensive pre‑registered tests on the bbob benchmark, AMM achieved the highest target share in 382 of 600 function‑dimension‑budget cells, outperforming up to seven competitors and maintaining strong performance across transformations and constraints.
arXiv:2504. 15610v4 Announce Type: replace Abstract: Fine-tuning a 7B language model for specialized advising is attractive in resource-constrained settings, but multi-epoch runs routinely exceed the wall-clock limits of the free-tier GPUs (Kaggle, Colab) such users rely on.
arXiv:2607. 00208v1 Announce Type: cross Abstract: Reinforcement learning for diffusion large language models (dLLMs) has largely moved to trajectory-aware methods.
The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.
arXiv:2605.06183v2 Announce Type: replace Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained...
arXiv:2605.29268v3 Announce Type: replace-cross Abstract: LLM-guided evolutionary search (Evolve systems) has reached state-of-the-art results on mathematical and combinatorial tasks, yet most existi...
arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.