← Back to all news
arXiv AI By Seyed Mohammad Asghari, Chris Chute, Vikranth Dwaracherla, Xiuyuan Lu, Mehdi Jafarnia, Victor Minden, Zheng Wen, Benjamin Van Roy

Efficient Exploration at Scale

Read the original on arXiv AI →

The paper presents an online learning algorithm that significantly boosts data efficiency for reinforcement learning from human feedback (RLHF). It incrementally updates reward and language models as choice data arrives, using a small affirmative nudge, an epistemic neural network for reward uncertainty, and information‑directed exploration. With Gemma LLMs, the method matches offline RLHF trained on 200K labels using fewer than 20K labels, achieving over a 10× improvement in data efficiency, and projects a 1,000× gain when scaled to 1M labels.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 15

Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

The paper introduces an exploration-guided prompt scaffolding framework for multimodal large language models, dynamically adjusting the prompt distribution during reinforcement learning post-training. It uses an Exploration Potential Score (EPS) derived from KL-regularized policy improvement to assess prompt utility without extra overhead, and a teacher model rewrites low-utility prompts to preserve intent while improving informativeness. Experiments on Geo3K, MMK12, MathVision, and MMMU-Pro show consistent performance gains, up to 9.7% in-domain and over 11% on out-of-distribution benchmarks.

By Yuanhao Yue, Qianli Ma, Chengyu Wang, Haoting Wang, Lei Shen, Jun Huang
llmsreinforcement-learningmultimodalbenchmarks
More like this →
arXiv AI
Jul 16

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

arXiv:2510. 11686v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.

By Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, Jordan T. Ash
llmsreinforcement-learning
More like this →
arXiv Machine Learning
Aug 27

Demystifying Reinforcement Learning Post-Training of Language Models

The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers. "whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."

By Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques
llmsnlpreinforcement-learningfine-tuning
More like this →