The paper introduces a new benchmark for diagrammatic reasoning in olympiad geometry, comprising 954 self‑contained problems and a 297‑problem hard subset. Each problem is paired with a human‑authored, high‑fidelity diagram in Asymptote code and a suite of metrics for evaluating diagram construction. Experiments show that current foundation models excel at solving the problems but produce markedly less faithful diagrams, with an average compile success rate of only 36.14%.
By Hsien Xin Peng, Anthony Kim, Alvin Li, Calvin Supasanya, Shivank Garg, Kevin Zhu
arXiv:2608. 12262v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration.
By Weihao Bo, Shan Zhang, Yanpeng Sun, Jie Liu, Yongke Yao, Jinhao Du, Wei He, Kai Zou, Zechao Li, Jingdong Wang
arXiv:2608. 08964v1 Announce Type: new Abstract: The generation of mathematically precise diagrams from tex- tual prompts has emerged as a critical yet underexplored capability of Large Language Models (LLMs).
By Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy, Sanyukta Tuti, Aryan Mistry
arXiv:2610.27626v1 Announce Type: new
Abstract: Multimodal Large Language Models show strong vision-language capabilities, yet their geometry-centric reasoning remains insufficiently understood. Exis...
By Jing Wang, Yitong Yang, Hao Dou, Yaomin Wu, Shuting He, Tingfei Shen, Jia Wu, Jiaqi Li, Yinglin Wang
SCAFFOLD is a large-scale structured dataset of computer science research figures, each paired with captions, context, questions, answers, and chain-of-thought reasoning traces. It contains 157,387 figure–question pairs from 3,058 arXiv papers, with subsets of 36,797 and 12,000 pairs for medium and small-scale use. The dataset was created using layout detection, PDF parsing, and AI-assisted question generation, and was used to benchmark a vision‑language model (Qwen2.5‑VL‑3B‑Instruct).
By Ranjit Raut, Aarav Subedi, Sagun Rai, Sudan Jha
HakushoBench is a Japanese chart and table visual question answering benchmark created from 33 governmental white papers, comprising 2,053 images across more than ten types. The dataset includes manually annotated and independently verified QA pairs that evaluate holistic understanding of charts and tables rather than just local visual cues. Experiments show that HakushoBench is significantly harder than existing Japanese benchmarks, with sub‑10B open‑weight models achieving at most 58.6% accuracy and even large models like Qwen3.5-397B-A17B lagging behind Gemini‑3‑Pro by 8.1 points.
By Issa Sugiura, Shuhei Kurita, Yusuke Oda, Naoaki Okazaki
MathVis is a new benchmark that tests large language models (LLMs) on their ability to generate and reason with mathematical figures. It contains 512 tasks, each with a mathematical statement, a reference diagram, executable TikZ or SVG code, a gold intermediate representation, atomic rubrics, and error codes. Initial results show that GPT‑6‑Astra, Claude Fable 5, and Gemini 3.1 Pro Preview achieve pass@1 scores of 31.25 %, 26.56 %, and 22.85 % respectively, yet the models tend to produce outputs dominated by formal languages and graphics programs rather than clear, diagram‑mediated reasoning, often resulting in visually confusing solutions for humans.
By Jingjing Wei, Hao Zhao, Xinke Li, Linxi Yu, Xiaosong Gao, Shanshan Xiao
PhysElite is a new bilingual multimodal benchmark designed to evaluate large language models on Olympiad-level physics problems. It contains 11,586 problems, each paired with visual diagrams, step-by-step bilingual Chinese‑English solution derivations, and the final answer. Benchmarking 18 models revealed that even the best reaches only 33.7% accuracy, and a step‑level analysis highlights where models falter in reasoning.
By Ruoran Xu, Wending Gao, Liyunfeng Chen, Aixin Shi, Haoyu Cheng, Zixiang Fang, Yiqiang Zou, Qiufeng Wang
arXiv:2607. 12982v1 Announce Type: new Abstract: Math reasoning has achieved significant progress with the rapid advancement of Multimodal Large Language Models (MLLMs), however analytic geometry remains largely underexplored, primarily due to the scarcity of annotated samples.
By Ruoran Xu, Wending Gao, Qiufeng Wang
arXiv:2608. 09281v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion.
By Chenxu Du, Kang An, Tengyue Wang, Zhongyu Yang, Xinqi Yang, Yuanchi Zhu, Hebao Zhu, Ziliang Wang, Faqiang Qian, Yunli Yang, Qibing Ren
The paper introduces a framework that combines a Geometric Vision Parser and a Symbolic Solver to enable a Large Language Model to solve complex plane geometry problems. By translating diagrams into symbolic representations and performing formal deductions, the approach reduces hallucinations and produces interpretable, human-like solutions. Experiments on a new benchmark from 2025 Chinese Zhongkao exams show performance comparable to Gemini 2.5 Pro.
By Weichen Dai, Rafael Medeiros Cabral, Ziyi Shou, Yan Cao, Xin Shen, Dongcai Lu, Yi Zhou
arXiv:2606. 04244v1 Announce Type: new Abstract: Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids.
By Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari, Yasamin Medghalchi, Ilker Hacihaliloglu, Mesrob Ohannessian, Lele Wang, Giuseppe Carenini