arXiv AI
Jun 6
Benchmarks in Leipzig
arXiv:2606. 05818v1 Announce Type: cross Abstract: Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers.
Benchmarks in Leipzig
Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the 3-day workshop *Benchmarks in Leipzig* with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany.
Related stories
arXiv AI
Jun 24
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.
arXiv AI
Aug 3
LLM Framework for Discovering Major Mathematical Conjectures: AI's Quest for the Next Riemann Hypothesis
arXiv:2607. 28632v1 Announce Type: new Abstract: Major mathematical conjectures still depend heavily on expert intuition, so a unified method for the systematic generation and validation of conjectures with substantial mathematical potential remains unavailable.
arXiv AI
Jun 15
MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis
arXiv:2606. 13782v1 Announce Type: new Abstract: Large Language Models (LLMs) have made notable progress in automated theorem proving, yet existing formal benchmarks remain limited in both mathematical coverage and difficulty.
arXiv Machine Learning
3d ago
The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
arXiv:2610. 02191v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions.
arXiv Computation and Language
Sep 15
Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
arXiv:2609.15145v1 Announce Type: new Abstract: The reasoning ability of large language models (LLMs) is a critical factor for practical LLM-based applications. To investigate the current reasoning c...
arXiv Computation and Language
6d ago
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
AdvancedMathBench is a new benchmark suite that evaluates large language models on advanced mathematical proof generation and verification. It includes ProverBench, with 245 problems from undergraduate to doctoral qualifying‑exam levels, and VerifierBench, which tests models’ ability to judge proof validity using 888 expert‑annotated trajectories. The suite features an automatic verification pipeline trained on expert data, and results show that even state‑of‑the‑art models perform poorly, highlighting a gap between generation and verification skills.
arXiv AI
Aug 25
More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
arXiv:2608.22048v1 Announce Type: new Abstract: Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy...
arXiv Machine Learning
Sep 11
Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification
The paper introduces HorizonMath, a benchmark of 113 largely unsolved mathematical problems across eight domains, paired with an open-source framework for automated verification. It focuses on the generator‑verifier gap, targeting problems that are hard to discover but easy to verify computationally, thereby avoiding costly formal proof verification or manual review. Using this framework, the authors found six novel solutions—three each from GPT‑5.4 Pro and GPT‑5.6 Sol—demonstrating that current models can contribute to mathematical research, while most state‑of‑the‑art models score below 10%.
arXiv AI
Sep 3
AI Mathematician: Towards Fully Automated Frontier Mathematical Research
The paper introduces the AI Mathematician (AIM) framework, which leverages Large Reasoning Models (LRMs) to tackle frontier mathematical research. AIM addresses the complexity and procedural rigor of research problems through an exploration mechanism for longer solution paths and a pessimistic reasonable verification method for reliability. Early experiments show AIM can autonomously construct significant proof components and uncover non‑trivial insights across real‑world mathematical topics.
arXiv AI
Sep 25
LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches
LiveMathematicianBench is a dynamic multiple‑choice benchmark for research‑level mathematical reasoning, built from recent arXiv papers published after model training cutoffs. It introduces a thirteen‑category logical taxonomy of theorem types and uses a proof‑sketch‑guided distractor pipeline to create plausible but invalid answer choices, enhancing sensitivity to genuine reasoning. Evaluation shows current large language models perform poorly, with the best model scoring 43.5% overall and only 17.6% under substitution‑resistant conditions, indicating the benchmark’s difficulty and realism.
arXiv AI
Jun 10
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
arXiv:2606. 10254v1 Announce Type: new Abstract: While Large Language Models (LLMs) have achieved near-perfect performance in \emph{solving} high-school mathematics, their ability to \emph{evaluate} the diverse reasoning processes of real human students remains under-examined.