arXiv AI October 31, 2026 By Tidor-Vlad Pricope

RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

RetroChessBench is a 12‑task benchmark designed to evaluate language models on retrograde chess reasoning, requiring the reconstruction of legal game histories and the distinction between possible and forced events. The benchmark includes exact‑length constructions, unique endings, exhaustive answer sets, references, rubrics, and a terminal‑reward environment. In tests, GPT‑6 Astra High failed to produce correct final answers in 22 of 24 attempts without tools and 18 of 24 with code execution, highlighting significant reasoning failures even in rule‑governed tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

Oct 29

RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

Frontier language models such as GPT 6 Astra have contributed to research mathematics, demonstrating substantial reasoning capabilities. Yet serious reasoning failures persist even in tasks governed b...

llms reinforcement-learning benchmarks

arXiv AI

Aug 7

Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

arXiv:2605. 12519v2 Announce Type: replace-cross Abstract: Training language models to produce both correct answers and sound reasoning remains an open challenge.

By Kyuyoung Kim, Kevin Wang, Yunfei Xie, Peiyang Xu, Peiyao Sheng, Chen Wei, Zhangyang Wang, Jinwoo Shin, Pramod Viswanath, Sewoong Oh

llms reinforcement-learning fine-tuning

arXiv AI

Jul 20

Understanding Reasoning from Pretraining to Post-Training

arXiv:2607. 16097v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it.

By Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov

llms reinforcement-learning fine-tuning

arXiv Computation and Language

Oct 22

Disentangling generalization and memorization in large language models using chess

arXiv:2601.16823v3 Announce Type: replace Abstract: Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine r...

By Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsaecker

llms

arXiv AI

Aug 18

Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

arXiv:2604. 06628v2 Announce Type: replace Abstract: A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes.

By Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, Yuejin Xie, Yafu Li, Quanshi Zhang, Xia Hu, Jing Shao, Dongrui Liu

llms reinforcement-learning fine-tuning safety data-engineering

arXiv Machine Learning

Jul 27

LeAct: Learning to Reason from Expert Actions

arXiv:2607. 21856v1 Announce Type: new Abstract: Modern reasoning models depend on reasoning data, today sourced from human annotations or distilled from stronger LLMs.

By Ziran Yang, Chengshuai Shi, Raj Ghugare, Benjamin Eysenbach, Karthik Narasimhan, Chi Jin

llms robotics benchmarks

One email a morning, machine-written

Related stories

RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

Understanding Reasoning from Pretraining to Post-Training

Disentangling generalization and memorization in large language models using chess

Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

LeAct: Learning to Reason from Expert Actions