arXiv Machine Learning September 25, 2026 By Noa Rubin, Zohar Ringel

RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory

Read the original on arXiv Machine Learning →

The paper investigates the optimization landscape of reinforcement learning with verifiable rewards (RLVR) on algorithmic tasks such as iterated group and quasigroup multiplication. By mapping entropy‑regularized RLVR with myopic tabular policies to a spin‑glass energy model, the authors prove that for a broad class of models with uncorrelated inputs the landscape is benign—there are no local minima that can trap training. They identify practical obstacles like diffusive barriers and gradient‑estimation errors, show these can be mitigated by choosing an appropriate entropy regulator, and demonstrate that a transformer trained from scratch with only last‑token rewards can learn an algorithmic chain of thought for iterated non‑Abelian group multiplications.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 30

On the Emergence of Implicit Curriculum in RLVR Learning Dynamics

arXiv:2602. 14872v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models.

By Yu Huang, Zixin Wen, Yuejie Chi, Yuting Wei, Aarti Singh, Yingbin Liang, Yuxin Chen

llms reinforcement-learning

arXiv AI

Jun 12

Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models

arXiv:2605. 17770v3 Announce Type: replace Abstract: The advancement of Large Reasoning Models (LRMs) has catalyzed a paradigm shift from reactive ``fast thinking'' text generation to systematic, step-by-step ``slow thinking'' reasoning, unlocking state-of-the-art performance in complex mathematical and logical tasks.

By Junyao Yang, Chen Qian, Kun Wang, Linfeng Zhang, Quanshi Zhang, Yong Liu, Dongrui Liu

reinforcement-learning benchmarks

arXiv AI

Jun 4

Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data

arXiv:2601. 15158v4 Announce Type: replace-cross Abstract: Transformers trained via Reinforcement Learning (RL) with outcome-based supervision can spontaneously develop the ability to generate intermediate reasoning steps (Chain-of-Thought).

By Yuval Ran-Milo, Yotam Alexander, Shahar Mendel, Nadav Cohen

llms reinforcement-learning

arXiv AI

Jul 7

rePIRL: Learn PRM with Inverse RL for LLM Reasoning

arXiv:2602. 07832v3 Announce Type: replace-cross Abstract: Process rewards have been widely used in deep reinforcement learning to improve training efficiency, reduce variance, and prevent reward hacking.

By Xian Wu, Kaijie Zhu, Ying Zhang, Lun Wang, Wenbo Guo

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 9

RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

arXiv:2511. 07317v2 Announce Type: replace-cross Abstract: We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs).

By Zhiyuan Zeng, Hamish Ivison, Yiping Wang, Lifan Yuan, Shuyue Stella Li, Zhuorui Ye, Siting Li, Jacqueline He, Runlong Zhou, Tong Chen, Chenyang Zhao, Yulia Tsvetkov, Simon Shaolei Du, Natasha Jaques, Hao Peng, Pang Wei Koh, Hannaneh Hajishirzi

llms reinforcement-learning benchmarks

arXiv AI

Sep 1

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

The paper investigates why reinforcement learning with verifiable rewards (RLVR) reduces the diversity of solutions in reasoning tasks. By analyzing the Countdown task, the authors show that RLVR contracts the solution space mainly at the entrance—before the first arithmetic operation—causing a 67% drop in solution coverage. They demonstrate that providing an unselected entrance prefix or applying entrance‑targeted interventions can restore or even improve coverage without harming accuracy.

By Qiancheng Zhou, Ruizhe Li

llms reinforcement-learning fine-tuning benchmarks

One email a morning, machine-written

Related stories

On the Emergence of Implicit Curriculum in RLVR Learning Dynamics

Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models

Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data

rePIRL: Learn PRM with Inverse RL for LLM Reasoning

RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space