Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
Read the original on arXiv AI →The paper investigates reinforcement learning with verifiable rewards (RLVR), where costly model-generated responses encourage larger rollout batches and a square‑root learning‑rate scaling. Using a two‑clock framework, the authors show that while the square‑root rule preserves first‑order drift per sample, larger batches suffer from curvature losses, leading to a critical batch size beyond which sample efficiency degrades. Experiments on Qwen2.5‑Math‑1.5B confirm the predicted progress gap and failure clock, and the authors propose a simple multi‑step RLVR schedule that starts below the critical batch and switches to a larger batch before failure.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.