The paper introduces GAC, a noise‑aware adaptive mixing controller that dynamically balances supervised fine‑tuning (SFT) and reinforcement learning (RL) gradients during hybrid post‑training of large language models. By deriving a closed‑form mixing weight that minimizes an MSE upper bound on the mixed gradient, GAC incorporates EMA smoothing, a schedule prior, and capped updates, all estimated online from training tensors. Experiments show that GAC outperforms rule‑based controllers, improving performance on diverse benchmarks by up to 3.8 percentage points, reducing KL‑drift by 28%, and cutting large mixing‑weight changes by over 70% with negligible overhead, with gains scaling from 1.5 B to 14 B parameters.
arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.
arXiv:2606. 25086v1 Announce Type: new Abstract: Many modern Language Model (LM) pipelines return an averaged model, such as an exponential moving average of the training iterates, rather than the final iterate itself.
The paper introduces COPC, a Coupled Off-Policy Correction method for asynchronous reinforcement learning of large language models. COPC coordinates policy-side and advantage-side corrections by combining token-level ratio masking with two-sided clipped-ratio weighting of TD residuals, addressing both policy mismatch and advantage staleness. Experiments show COPC outperforms existing asynchronous baselines on tool-integrated mathematical reasoning and search tasks, while maintaining training stability and minimal overhead.
The paper introduces WHALE, a method that alternates between updating a language model’s weights and searching for a better harness (the code that manages context and control flow). By iteratively fine‑tuning the model under the current harness and then optimizing the harness under the updated model, WHALE improves performance across search QA, math reasoning, and chess puzzles, outperforming weight‑only, harness‑only, and Fast‑Slow Training by 4.15–24.38 percentage points in mean@8 accuracy. The approach uses either fixed phase lengths or an adaptive patience rule to decide when to switch phases, and the authors provide code on GitHub.
GrowMTP is a method that trains a draft head entirely within the reinforcement learning (RL) loop, using supervision from the RL verification step and a rollout distribution that is narrower than pretraining. By detaching draft‑head updates from the policy backbone, it enables online training of the draft head from scratch. Experiments on Qwen3‑4B, MiMo‑7B‑SFT, and Qwen3.5‑4B‑Base show rollout speedups ranging from 1.36× to 2.13× and overall end‑to‑end speedups from 1.20× to 1.60×, making it a modular acceleration component for RL frameworks lacking pretrained draft heads.
arXiv:2606. 04847v1 Announce Type: cross Abstract: Native GPU kernel generation turns high-level tensor programs into executable, efficient low-level code.
EfficientTDMPC is a sample‑efficient, model‑based reinforcement learning method for continuous control that builds on the TD‑MPC family. It reduces estimation error by using an ensemble of dynamics models and averaging return estimates across models and rollout depths, and it can penalize uncertainty in the planner objective. Practical improvements such as fresher buffer data and reduced compute enable the algorithm to benefit from a higher update‑to‑data ratio, achieving state‑of‑the‑art sample efficiency on HumanoidBench‑Hard and DMC hard while matching state‑of‑the‑art on DMC easy.
arXiv:2609.36830v1 Announce Type: new Abstract: Fully asynchronous reinforcement learning (RL) improves resource utilization in large language model post-training by overlapping rollout generation wi...
The paper introduces Solver-Gradient Guided Reinforcement Learning (SG‑RL), a method that augments standard RL with bounded gradients from a differentiable MPC solver to adapt cost‑function weights online. SG‑RL integrates solver‑gradient guidance into PPO through actor‑update scaling, policy loss, advantage estimation, and value‑function learning, achieving comparable or superior closed‑loop performance while requiring up to 70.6% fewer samples. Experiments on two autonomous racing platforms with intentional model mismatch demonstrate that SG‑RL outperforms both RL and gradient‑based policy learning baselines and generalizes zero‑shot to unseen environments.
arXiv:2505. 23878v2 Announce Type: replace-cross Abstract: Optimizing pretraining data composition is pivotal for LLM generalization.