arXiv AI
Sep 28

Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design

The paper examines how evaluation design can cause significant fluctuations in the benchmark results of reasoning models, particularly the Deepseek‑R1‑Distill series. It shows that subtle changes in evaluation conditions lead to large variations in reported performance, a phenomenon also seen in other open‑source models fine‑tuned from Deepseek‑R1‑Distill and in the QwQ‑32B model. The authors call for a more rigorous evaluation paradigm and provide empirical assessments of the Deepseek‑R1‑Distill models.

By Yongfu Zhu, Lin Sun, Jinzhu Wu, Weihong Lin, Xiaoqi Jian, Guangxiang Zhao, Change Jia, Linglin Zhang, Sai-er Hu, Yuhan Wu, Xiangzheng Zhang