A Sign Law for Entropy Drift under Sampler Mismatch in RLVR

The paper investigates how entropy behaves in reinforcement learning with verifiable rewards (RLVR) when the sampling method used during rollouts differs from the learner’s policy. It shows that a constant advantage shift alters the entropy of a softmax policy in a way that depends on the sampler: positive shifts sharpen the policy for samplers whose likelihood ratio increases with token probability (e.g., top‑k, top‑p, min‑p, and temperatures below one) and flatten it for flatter samplers. In group‑based RLVR with token‑level losses, the entropy drift is governed by the token‑weighted net advantage and a covariance term that can amplify over long horizons, with empirical results on 2B and 9B models demonstrating how different sampling and baseline strategies affect entropy stability and held‑out accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• reinforcement-learning