arXiv AI

Math Reasoning in LLMs is Organized by Approach, Not Topic

The paper argues that large language models (LLMs) organize their internal mathematical reasoning by reusable reasoning approaches rather than by the benchmark topics they are tested on. Using a generation‑replay protocol, the authors extract activation‑importance signatures from eight models across five math sources, cluster these signatures, and find that the resulting groups align more closely with reasoning approaches than with topics. The study shows that changing the requested reasoning approach shifts cluster assignments, while paraphrasing the prompt does not, underscoring the primacy of approach over topic in LLM reasoning.

arXiv AI

Sep 25

LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches

LiveMathematicianBench is a dynamic multiple‑choice benchmark for research‑level mathematical reasoning, built from recent arXiv papers published after model training cutoffs. It introduces a thirteen‑category logical taxonomy of theorem types and uses a proof‑sketch‑guided distractor pipeline to create plausible but invalid answer choices, enhancing sensitivity to genuine reasoning. Evaluation shows current large language models perform poorly, with the best model scoring 43.5% overall and only 17.6% under substitution‑resistant conditions, indicating the benchmark’s difficulty and realism.

By Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, Nima Mesgarani

llms benchmarks

arXiv AI

Jul 8

Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

arXiv:2605. 19723v2 Announce Type: replace-cross Abstract: Mathematical reasoning is essential for problem-solving in education, science, and industry, serving as a crucial benchmark for evaluating artificial intelligence systems.

By Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad, Mehwish Fatima

llms fine-tuning benchmarks

arXiv Machine Learning

Oct 3

The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models

arXiv:2610. 02191v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions.

By Shuo Xing, Zilin Dai, Chengyuan Qian, Fangzhou Lin, Wenjing Chen, Ping He, Pan Lu, Alvaro Velasquez, Mohit Bansal, Zhengzhong Tu

llms efficiency benchmarks

arXiv AI

Jun 8

A Comprehensive Anatomy of Human and DeepSeek-R1 LLM Mathematical Reasoning

arXiv:2606. 07410v1 Announce Type: cross Abstract: The emergence of "Aha moments" in large language models, particularly DeepSeek-R1-0120, has raised the question of whether these systems genuinely reason or merely imitate the appearance of reasoning.

By Yuxiang Chen, Jun Wang

llms

arXiv AI

Jun 10

RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning

arXiv:2606. 10254v1 Announce Type: new Abstract: While Large Language Models (LLMs) have achieved near-perfect performance in \emph{solving} high-school mathematics, their ability to \emph{evaluate} the diverse reasoning processes of real human students remains under-examined.

By Yiteng Mao, Kenan Xu, Yijia Lyu, Wenhao Li, Jianlong Chen, Xiangfeng Wang

llms rag benchmarks

arXiv AI

Jul 8

PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.

By Daryna Dementieva, Nikolay Babakov, Kathy H\"ammerl, Ilseyar Alimova, Jind\v{r}ich Libovick\'y, Shu Okabe, Miras Baisbay, Lukas Edman, Abrorkhon Inomkhujaev, Antonia Karamolegkou, Mateusz Lango, Volkan \"Ozer, Nikola Selic, Subhankar Swain, Tsedeniya Kinfe Temesgen, Galit Bary Weisberg, Alexander Fraser

llms benchmarks

arXiv AI

Jun 10

Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning

arXiv:2606. 09873v1 Announce Type: cross Abstract: Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer.

By Aditya Sharma, Christopher J. Pal, Amal Zouaq

llms rag multimodal benchmarks

arXiv AI

Jul 21

Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering

arXiv:2607. 18100v1 Announce Type: new Abstract: Extended reasoning has become standard for frontier Large Language Models (LLMs), yet the trajectories these models produce remain largely uncontrollable.

By Sheldon Yu, Tong Yu, Xunyi Jiang, Rohan Surana, Gagan Mundada, Sungchul Kim, Lina Yao, Julian McAuley, Junda Wu

llms

arXiv AI

Oct 9

An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems

The paper introduces GAP, a methodology that automatically generates mathematically equivalent variants of existing math problems using surface renames and kernel rewrites. Applying GAP to all 1,051 William Lowell Putnam Competition problems creates PutnamGAP, a 6,306-item dataset with 5,255 unseen variants. Evaluation of 18 commercial and open-source LLMs shows accuracy drops across all models, especially under kernel rewrites, indicating a persistent weakness in transferring proof plans to new mathematical settings.

By Yuren Hao, Xiang Wan, ChengXiang Zhai

llms benchmarks

arXiv AI

Oct 21

CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning

arXiv:2603.00889v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, largely enabled by supervised fine-tuning (SFT)- and reinforce...

By Xinyu Zhu, Yihao Feng, Yanchao Sun, Xianzhi Du, Pingzhi Li, Olli Saarikivi, Yun Zhu, Yu Meng

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 9

MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning

arXiv:2607. 07391v1 Announce Type: new Abstract: Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue.

By Charbel Al Bateh, Samer Saab Jr

llms benchmarks

arXiv Computation and Language

Sep 23

Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models

The paper demonstrates that frontier language models can be prompted to expose their internal chain-of-thought reasoning via a simple custom tool. By comparing these extracted traces to native reasoning on open-source models, the authors confirm that the externalized reasoning aligns with genuine reasoning and outperforms no-reasoning baselines across math, science, and code tasks. They further analyze the structure of the reasoning, noting token-efficient, directed reasoning in models like GPT‑6 Astra, which externalizes only crucial steps while handling elementary ones internally.

By Xiaoyu Luo, Tao Ren, Wenrui Yu, Xiao Li, Qiongxiu Li, Johannes Bjerva

llms benchmarks

Related stories