arXiv:2603. 13191v2 Announce Type: replace-cross Abstract: While large language models (LLMs) have transformed AI agents into proficient executors of computational materials science, performing a hundred simulations does not make a researcher.
arXiv:2503.21821v2 Announce Type: replace Abstract: We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended prob...
arXiv:2605.26087v2 Announce Type: replace-cross Abstract: Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of...
The paper introduces HorizonMath, a benchmark of 113 largely unsolved mathematical problems across eight domains, paired with an open-source framework for automated verification. It focuses on the generator‑verifier gap, targeting problems that are hard to discover but easy to verify computationally, thereby avoiding costly formal proof verification or manual review. Using this framework, the authors found six novel solutions—three each from GPT‑5.4 Pro and GPT‑5.6 Sol—demonstrating that current models can contribute to mathematical research, while most state‑of‑the‑art models score below 10%.
PhysElite is a new bilingual multimodal benchmark designed to evaluate large language models on Olympiad-level physics problems. It contains 11,586 problems, each paired with visual diagrams, step-by-step bilingual Chinese‑English solution derivations, and the final answer. Benchmarking 18 models revealed that even the best reaches only 33.7% accuracy, and a step‑level analysis highlights where models falter in reasoning.
The paper introduces SCOPE, a framework that enables multiple scientist agents to pursue conjecture-driven research trajectories by modeling equations, formulating questions, and conducting active experiments, then sharing evidence to refine their conjectures. It also presents ClosureBench, an interactive benchmark of 60 closure-discovery tasks across five scientific domains, designed to evaluate equation discovery beyond accuracy by measuring evidence faithfulness, mechanism insight, and experimental effectiveness. Experiments show that SCOPE outperforms diverse baselines on both ClosureBench and a real-world scientific discovery case, yielding more accurate, reliable, and effective scientific discoveries.
arXiv:2604.18936v2 Announce Type: replace Abstract: Despite the growing application of large language models (LLMs) to theoretical physics, there has been little academic exploration of how domain-sp...
SciGame is a new benchmark that transforms scientific literature into structured cards, each providing both an Original and a Counterfactual environment for the same task. It evaluates language‑model agents in identification and discovery scenarios under Prior‑only, Fixed, and Adaptive protocols, using historical canonical laws and a live temporal collection. Experiments with five advanced models reveal that prior knowledge can hurt performance when relations change, adaptive experimentation benefits strong models but not weak ones, and accurate explanations may mask quantitative errors, highlighting the difficulty of recovering exact symbolic laws.
arXiv:2512. 19799v2 Announce Type: replace Abstract: Advances in LLM reasoning and tool use have enabled agentic science, yet frontier theoretical and computational physics remains challenging because research requires deep domain expertise, long-horizon reasoning, and reliable numerical computation.
PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.
The paper introduces SCILAWS-BENCH, a benchmark for evaluating large language models (LLMs) on scientific law discovery. It contains 118 problems from 381 scientific papers, covering 291 candidate laws and about 8 million real data points across six disciplines. The benchmark offers two settings: SCILAWS-REAL, where models must propose laws from fixed real observations, and SCILAWS-PARALLEL, where models actively query synthetic worlds to recover hidden laws.
arXiv:2604. 19341v2 Announce Type: replace-cross Abstract: Scientific discovery often requires many cycles of proposing, testing, and refining candidate solutions.