arXiv Computation and Language
Sep 23
FrontierMath Erd\H{o}s
arXiv:2609. 25050v1 Announce Type: new Abstract: We introduce FrontierMath Erd\H{o}s (FME), a benchmark of 68 Erd\H{o}s problems that are open as of August 2026.
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
OpenProblemBench is a new benchmark that presents 82 unresolved problems from mathematics and theoretical physics, each with detailed context, assumptions, and prior progress. The benchmark evaluates AI submissions on correctness, completeness, and progress using four independent evaluator models, without providing reference solutions. GPT‑6‑Astra achieved the highest mean judged solve rate of 14.0%, outperforming other evaluated models, and the study links stronger outcomes to improved problem representation, general reasoning, and proof steps.
Related stories
arXiv Machine Learning
Sep 11
Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification
The paper introduces HorizonMath, a benchmark of 113 largely unsolved mathematical problems across eight domains, paired with an open-source framework for automated verification. It focuses on the generator‑verifier gap, targeting problems that are hard to discover but easy to verify computationally, thereby avoiding costly formal proof verification or manual review. Using this framework, the authors found six novel solutions—three each from GPT‑5.4 Pro and GPT‑5.6 Sol—demonstrating that current models can contribute to mathematical research, while most state‑of‑the‑art models score below 10%.
arXiv AI
Oct 31
OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery
OpenFrontiers is a cross‑disciplinary resource that compiles 12,151 open research problems and a benchmark of 200 recently resolved mathematics problems, including 110 solved in 2026 and 90 still open. Each benchmark problem is formally stated in Lean 4, verified by automated probes, and reviewed independently. The evaluation protocol tests LLM agents under closed‑book, frozen‑retrieval, and live‑web conditions, revealing that closed‑book rediscovery is rare while live‑web success largely reflects retrieval of published solutions.
arXiv AI
Aug 19
ASI-Bench: At the Dawn of Artificial Superintelligence
ASI‑Bench is a new benchmark that evaluates AI systems on their ability to conduct innovative exploration and autonomous scientific research across 11 domains, using 60 project‑level tasks. It progressively removes human methodological guidance to test whether AI can independently select methods, execute research, and produce verifiable results. Results from 18 state‑of‑the‑art agent–model configurations show a sharp performance drop when guidance is reduced, indicating current systems still rely heavily on human input.
arXiv AI
Aug 3
Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review
arXiv:2607. 28631v1 Announce Type: new Abstract: AI Scientist systems capable of autonomous research have the potential to significantly accelerate scientific discovery.
arXiv AI
Oct 31
Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures
The paper introduces OpenConjecture, a dataset of mathematical conjectures from arXiv papers since February 2026, and evaluates frontier GPT models on 200 randomly sampled conjectures. The models generate candidate solutions for about 60% of the conjectures, with counterexamples occurring roughly 1.3 times more often than proofs. The study examines solution novelty, citation practices, and fidelity to the original statements, finding that some solutions deviate from authors’ intent and that many proofs closely follow existing results, though some exhibit moderate novelty.
OpenAI Blog
Dec 16, 2025
Evaluating AI’s ability to perform scientific research tasks
OpenAI introduces FrontierScience, a benchmark testing AI reasoning in physics, chemistry, and biology to measure progress toward real scientific research.
arXiv AI
Oct 21
ReuF2F: Benchmarking Coding Agents on Formalized Undergraduate Research Problems
ReuF2F is a Lean 4 benchmark comprising 244 formalized open problems sourced from Research Experiences for Undergraduates (REU). It aims to evaluate coding agents on less prominent, entry‑level research problems across diverse mathematical subfields, using a Docker‑based Lean project setup and a unified harness called Leanimum‑agent. The study finds that even the best model solves only about 40 % of the problems, indicating significant room for improvement.
arXiv AI
Oct 31
Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs
The paper reports on the Ramanujan Challenge, a benchmark of research‑level number theory problems, and evaluates submissions from AI models, CAS, and formal tools. It highlights that some correct solutions differ conceptually from official ones and even proves a published open conjecture. The authors introduce an AI‑assisted evaluation pipeline that significantly lowers false positives in natural language proofs and improves accuracy for supporting code, offering a scalable method for assessing high‑level mathematical work.
OpenAI Blog
Oct 6
Sharing AI progress in mathematics
OpenAI has released new results on open problems in mathematics derived from its internal frontier model. The company is sharing Lean proof formalizations and detailed research findings on GitHub. This effort makes advanced mathematical progress publicly accessible and encourages collaboration within the research community.
arXiv AI
Aug 14
PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Research
arXiv:2512. 19799v2 Announce Type: replace Abstract: Advances in LLM reasoning and tool use have enabled agentic science, yet frontier theoretical and computational physics remains challenging because research requires deep domain expertise, long-horizon reasoning, and reliable numerical computation.
arXiv Computation and Language
Oct 26
PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research
arXiv:2610.32575v1 Announce Type: new Abstract: Large language models and agents have shown significant power in advanced physical research. In this paper, We introduce PHYRD, a living benchmark for...