arXiv AI

Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures

The paper introduces OpenConjecture, a dataset of mathematical conjectures from arXiv papers since February 2026, and evaluates frontier GPT models on 200 randomly sampled conjectures. The models generate candidate solutions for about 60% of the conjectures, with counterexamples occurring roughly 1.3 times more often than proofs. The study examines solution novelty, citation practices, and fidelity to the original statements, finding that some solutions deviate from authors’ intent and that many proofs closely follow existing results, though some exhibit moderate novelty.

arXiv AI
Sep 25

Learning to Discover Interesting Mathematics

The paper introduces a method for evaluating the intrinsic interestingness of mathematical theorems by comparing the length of their proofs to the length of their statements. It trains a 27B language model to predict proof difficulty, enabling the generation and selection of more interesting theorems while significantly reducing overlap with existing Mathlib. The approach allows iterative expansion of a self‑building, machine‑verified mathematical library guided by quantifiable metrics.

By Niket Patel, Ahmad Rammal, Amaury Hayat, Remi Munos, Julia Kempe
arXiv Machine Learning
Sep 11

Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification

The paper introduces HorizonMath, a benchmark of 113 largely unsolved mathematical problems across eight domains, paired with an open-source framework for automated verification. It focuses on the generator‑verifier gap, targeting problems that are hard to discover but easy to verify computationally, thereby avoiding costly formal proof verification or manual review. Using this framework, the authors found six novel solutions—three each from GPT‑5.4 Pro and GPT‑5.6 Sol—demonstrating that current models can contribute to mathematical research, while most state‑of‑the‑art models score below 10%.

By Erik Y. Wang, Sumeet R. Motwani, James V. Roggeveen, Eliot Hodges, Dulhan Jayalath, Charles London, Kalyan Ramakrishnan, Jakob Foerster, Cheng Zhang, Flaviu Cipcigan, Philip Torr, Alessandro Abate
arXiv AI
Oct 9

An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems

The paper introduces GAP, a methodology that automatically generates mathematically equivalent variants of existing math problems using surface renames and kernel rewrites. Applying GAP to all 1,051 William Lowell Putnam Competition problems creates PutnamGAP, a 6,306-item dataset with 5,255 unseen variants. Evaluation of 18 commercial and open-source LLMs shows accuracy drops across all models, especially under kernel rewrites, indicating a persistent weakness in transferring proof plans to new mathematical settings.

By Yuren Hao, Xiang Wan, ChengXiang Zhai
arXiv AI
Jun 10

ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics

arXiv:2606. 10479v1 Announce Type: new Abstract: Combinatorics is central to Olympiad-level mathematical problem solving, requiring deep discrete reasoning, creative constructions, and rigorous structural insight.

By Shunkai Zhang, Haoran Zhang, Yun Luo, Qianjia Cheng, Haodi Lei, Yizhuo Li, Runzhe Zhan, Zhilin Wang, Bangjie Xu, Yucheng Su, Xinmiao Han, Xiaoye Qu, Dongrui Liu, Zhouchen Lin, Yu Qiao, Ning Ding, Yafu Li, Yu Cheng
arXiv AI
Aug 3

LLM Framework for Discovering Major Mathematical Conjectures: AI's Quest for the Next Riemann Hypothesis

arXiv:2607. 28632v1 Announce Type: new Abstract: Major mathematical conjectures still depend heavily on expert intuition, so a unified method for the systematic generation and validation of conjectures with substantial mathematical potential remains unavailable.

By Alizer Wong, Zixin Zeng, Yi Tan, Wenyuan Li, Xuhang Chen, Xingru Lai, Yang Shi, Liangsi Lu, Yanhui Chen
arXiv Computation and Language
Aug 27

MathAdv: What Theorem Provers Know, Reason, Formalize, and Generalize

MathAdv is a diagnostic benchmark for formal theorem proving that covers 13 undergraduate- and graduate-level mathematics domains. It includes Lean 4 proofs and up to three auxiliary tasks—multiple-choice questions, fill-in-the-blank problems, and expert-crafted transformations—to probe knowledge, informal reasoning, and robustness to problem presentation. Evaluation of current theorem provers shows formalization is a major bottleneck, performance varies by domain, natural-language guidance can help or hinder models, and equivalent reformulations reveal significant robustness gaps.

By Jiaxin Yuan, Connor Martinez Lockhart, Xiaoyu Liu, Jiaqi Wang, Chenghao Deng, Xiayimei Han, Vlasios Mastrantonis, Dmitrii Gudin, Shaopeng Zhu, Abdirisak Abdullahi Mohamed, Bilal Hamdi Aytekin, Jiewen Lang, Zezheng Song, Furong Huang