arXiv Machine Learning
Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problems
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 5
Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
arXiv:2512. 19905v3 Announce Type: replace-cross Abstract: Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time.
arXiv AI
Oct 1
Revisiting scaling laws for reward optimization
The paper presents a new scaling law for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}), where M is the number of preference comparisons used to train a proxy reward model and K is the KL‑divergence budget relative to a reference policy. The authors derive this law using an information‑theoretic model, prove its tightness, and validate it with extensive experiments involving a 70B gold reward model and smaller proxy models (0.6B–4B). The empirical results demonstrate a strong fit (R² 97–99 %) across different model sizes, noise levels, and optimization methods, suggesting that reward optimization behaves like a simple selection task over IID Gaussian variables with noisy feedback.
arXiv Computation and Language
Sep 1
Small Reward Models via Backward Inference
arXiv:2602.13551v3 Announce Type: replace Abstract: Reward models (RMs) play a central role throughout the language model (LM) pipeline, particularly in non-verifiable domains. However, the dominant...
arXiv AI
Oct 9
How to post-train on a surrogate: Envelope sampling mitigates reward hacking
The paper addresses reward hacking that arises when large language models (LLMs) are post‑trained using cheap surrogate judges instead of expensive human feedback. It introduces envelope sampling, a theoretically grounded recalibration technique that uses a small set of ground‑truth annotated outputs to adjust the surrogate judge before further training. Experiments on clinical note generation and a sycophancy task demonstrate that envelope sampling reduces reward hacking compared to traditional base‑model recalibration methods.
arXiv Machine Learning
Jun 9
Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verification
arXiv:2605. 17609v2 Announce Type: replace Abstract: Many inference-time language-model pipelines combine a cheap reward signal with an expensive verifier, such as exact answer checking in mathematical reasoning or hidden-test execution in code generation.
arXiv AI
Aug 25
Safety Hacking in Constrained Best-of-$N$ Inference-time Scaling
arXiv:2608. 22915v1 Announce Type: cross Abstract: Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward.