NL2AGBench is a benchmark that evaluates how well large language models can translate English geometry problems into the formal language required by AlphaGeometry’s theorem‑proving engine. The study tests ten state‑of‑the‑art LLMs, comparing executable translation accuracy, syntactic correctness, and error types, and finds a large gap between closed‑source and open‑source models. The authors also propose an error taxonomy and test mitigation strategies such as few‑shot prompting, fine‑tuning, and human‑guided hinting, which improve performance across model families.
By Samuel Xiao, Judy Song, Rory Hu, Ziliang Zong
The paper introduces an epistemically and formally grounded ensemble (EFG) of large language model judges to evaluate autoformalization tasks in formal mathematics. It defines four criteria—logical preservation, mathematical consistency, formal quality, and formal validity—to provide a transparent, multi‑granular assessment. Experiments show that this ensemble outperforms coarse‑grained models, offering a scalable and interpretable proxy for evaluating formal mathematical reasoning.
By Lan Zhang, Marco Valentino, Jordan Meadows, Andre Freitas
arXiv:2512.09874v3 Announce Type: replace-cross
Abstract: Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from...
By Pius Horn, Janis Keuper
arXiv:2606. 16003v1 Announce Type: new Abstract: This work investigates the ability of large language models (LLMs) to generate mathematical equations from scientific texts.
By Yifan Mo, Xiao Fu, Yue Su, Qingyu Meng, Koen Hindriks, Qingzhi Liu, Jiahuan Pei
arXiv:2605. 19723v2 Announce Type: replace-cross Abstract: Mathematical reasoning is essential for problem-solving in education, science, and industry, serving as a crucial benchmark for evaluating artificial intelligence systems.
By Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad, Mehwish Fatima
arXiv:2510. 04520v2 Announce Type: replace Abstract: Accurate auto-formalization of theorem statements is essential for advancing automated discovery and verification of research-level mathematics, yet remains a major bottleneck for LLMs due to hallucinations, semantic mismatches, and their inability to synthesize new definitions.
By Hanyu Wang, Ruohan Xie, Yutong Wang, Guoxiong Gao, Xintao Yu, Bin Dong