arXiv Machine Learning
Metropolis-Hastings Dominates Importance Resampling for Policy Composition
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 7
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
Hugging Face Trending Papers
Jul 6
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.
arXiv Machine Learning
Jun 25
Learning with a Single Rollout via Monte Carlo Pass@k Critic
arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.
arXiv AI
Jun 15
Rethinking the Trust Region in LLM Reinforcement Learning
arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
arXiv Machine Learning
6d ago
Efficient Best-of-N policy evaluation for inference-time alignment
The paper introduces a sample‑only framework for evaluating and selecting Best‑of‑N (BoN) inference‑time alignment policies without needing response likelihoods. By exploiting the order‑statistic structure of BoN, the authors express required density ratios through score‑rank probabilities that can be estimated from samples alone, and develop a doubly robust estimator (BoN‑DR) that reuses a shared auxiliary sample pool across candidate budgets. The method provides valid asymptotic inference even with reward estimator misspecification, proves efficiency, and offers two selection rules—maximizing estimated policy value or a lower confidence bound—to avoid over‑optimization and guarantee no harm, as demonstrated on synthetic data and GSM8K.
arXiv Machine Learning
Aug 10
Inference-Time Scaling of Diffusion Language Models via Trajectory Refinement
arXiv:2507. 08390v5 Announce Type: replace Abstract: Discrete diffusion models have recently emerged as strong alternatives to autoregressive language models, matching their performance through large-scale training.