OpenAI Blog May 31, 2023

Improving mathematical reasoning with process supervision

We've trained a model to achieve a new state-of-the-art in mathematical problem solving by rewarding each correct step of reasoning (“process supervision”) instead of simply rewarding the correct final answer (“outcome supervision”). In addition to boosting performance relative to outcome supervision, process supervision also has an important alignment benefit: it directly trains the model to produce a chain-of-thought that is endorsed by humans.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at OpenAI Blog.

arXiv AI

Jun 6

Controllable and Verifiable Process Data Synthesis for Process Reward Models

arXiv:2605. 02395v2 Announce Type: replace Abstract: Process reward models (PRMs) rely on high-quality process supervision data, yet existing construction methods often provide limited control over error location, error type, and trajectory consistency.

By Yinghui Chi, Lucien Wang

rag benchmarks

arXiv AI

Oct 1

Learning Process Rewards via Reasoning State Propagation

The paper introduces Reasoning State Propagation (RSP), a method that models each reasoning prefix with a binary validity state and learns transitions between successive states. RSP predicts break and repair probabilities to connect intermediate reasoning states to the final outcome, enabling outcome supervision to guide learning of earlier steps. Experiments on reasoning search, response selection, and reinforcement learning show RSP consistently outperforms existing Process Reward Models, achieving notable gains over Qwen2.5-Math-PRM.

By Kai Gan, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, Tong Wei

reinforcement-learning

arXiv AI

Sep 30

Inducing Process Supervision from Outcome-Only Reinforcement Learning

arXiv:2609.36641v1 Announce Type: cross Abstract: Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning....

By Shengda Fan, Xin Cong, Zhong Zhang, Haotian Chen, Yankai Lin

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 9

Learning to Plan by Looking Back: Hindsight Hierarchies for Training Reasoning Models

The paper proposes a self‑improvement loop for reasoning models that leverages hindsight: when a problem is too hard, a supplied solution can help the model learn useful solution ideas. The method trains a single model to (1) predict solution ideas from problems alone, (2) reverse‑engineer ideas from problems and known solutions, and (3) solve problems using provided ideas. The loop alternates between reverse‑engineering ideas from problems with supplied solutions and using these ideas as additional supervision for joint training of all three capabilities, with a formal specification and an instantiation for interactive theorem proving in Lean.

By Lars Simon, Holger Eble, Manuel Radons

arXiv Computation and Language

Sep 30

RAWR: Reward Assignment Without Rollouts in Verifiable Domains

arXiv:2603.17815v2 Announce Type: replace Abstract: Understanding and evaluating multi-step reasoning in LLMs at the level of individual steps remains a key challenge. Process reward models (PRMs) pr...

By Corentin Royer (International Business Machines), Anna Hedstr\"om (ETH AI Center), Debarun Bhattacharjya (Lirio), Gaetano Rossiello (International Business Machines), Andrea Giovannini (International Business Machines), Mennatallah El-Assady (Department of Computer Science, ETH Zurich)

llms benchmarks data-engineering

arXiv AI

5d ago

Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning

arXiv:2604.02341v2 Announce Type: replace-cross Abstract: Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final...

By Mohammad Rezaei, Jens Lehmann, Sahar Vahdati

llms reinforcement-learning safety

One email a morning, machine-written

Related stories

Controllable and Verifiable Process Data Synthesis for Process Reward Models

Learning Process Rewards via Reasoning State Propagation

Inducing Process Supervision from Outcome-Only Reinforcement Learning

Learning to Plan by Looking Back: Hindsight Hierarchies for Training Reasoning Models

RAWR: Reward Assignment Without Rollouts in Verifiable Domains

Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning