← Back to all news
arXiv AI By Yuelin Hu, Wei Liu, Zhenbo Yu, Zhengxue Cheng, Li Song

GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

Read the original on arXiv AI →

The paper introduces GAC, a noise‑aware adaptive mixing controller that dynamically balances supervised fine‑tuning (SFT) and reinforcement learning (RL) gradients during hybrid post‑training of large language models. By deriving a closed‑form mixing weight that minimizes an MSE upper bound on the mixed gradient, GAC incorporates EMA smoothing, a schedule prior, and capped updates, all estimated online from training tensors. Experiments show that GAC outperforms rule‑based controllers, improving performance on diverse benchmarks by up to 3.8 percentage points, reducing KL‑drift by 28%, and cutting large mixing‑weight changes by over 70% with negligible overhead, with gains scaling from 1.5 B to 14 B parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Aug 4

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.

By Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri
benchmarks
More like this →
arXiv Machine Learning
Oct 8

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

The paper introduces COPC, a Coupled Off-Policy Correction method for asynchronous reinforcement learning of large language models. COPC coordinates policy-side and advantage-side corrections by combining token-level ratio masking with two-sided clipped-ratio weighting of TD residuals, addressing both policy mismatch and advantage staleness. Experiments show COPC outperforms existing asynchronous baselines on tool-integrated mathematical reasoning and search tasks, while maintaining training stability and minimal overhead.

By Zicheng Hu, Zhijian Zhou, Xuan Zhang, Yuchen Liu, Cheng Chen, Yuan Li, Qi Gu, Yan Feng, Hongyan Hao, Chao Qu
llmsreinforcement-learningsafety
More like this →
arXiv AI
Sep 2

WHALE: A Simple Recipe for Joint Harness-Weight Optimization

The paper introduces WHALE, a method that alternates between updating a language model’s weights and searching for a better harness (the code that manages context and control flow). By iteratively fine‑tuning the model under the current harness and then optimizing the harness under the updated model, WHALE improves performance across search QA, math reasoning, and chess puzzles, outperforming weight‑only, harness‑only, and Fast‑Slow Training by 4.15–24.38 percentage points in mean@8 accuracy. The approach uses either fixed phase lengths or an adaptive patience rule to decide when to switch phases, and the authors provide code on GitHub.

By Haechan Kim, Yoonho Lee, Gisang Lee, Chelsea Finn, Kangwook Lee
agentsnlpfine-tuning
More like this →