arXiv Machine Learning

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

The paper introduces CodeRQ-Bench, the first benchmark for assessing large language model reasoning quality across coding tasks such as generation, summarization, and classification. It analyzes over a thousand mismatches from existing evaluators, identifies recurring limitations, and derives design insights that lead to a new two‑stage evaluator, VERA. Experiments show VERA outperforms strong baselines, improving AUCROC by up to 0.26 and AUPRC by up to 0.21 on four datasets.

arXiv Machine Learning

Jun 16

Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification

arXiv:2601. 22642v2 Announce Type: replace Abstract: Large Language Models (LLMs) show remarkable capabilities, yet their stochastic next-token prediction creates logical inconsistencies and reward hacking that formal symbolic systems avoid.

By Chuxue Cao, Jinluan Yang, Haoran Li, Kunhao Pan, Zijian Zhao, Zhengyu Chen, Yuchen Tian, Lijun Wu, Conghui He, Sirui Han, Yike Guo

llms fine-tuning benchmarks

arXiv AI

Aug 3

RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment

arXiv:2507. 22580v2 Announce Type: replace-cross Abstract: Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention.

By Marcos Fuster-Pena, David de-Fitero-Dominguez, Antonio Garcia-Cabot, Eva Garcia-Lopez

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 10

CodeAlchemy: Synthetic Code Rewriting at Scale

arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.

By Ankit Gupta, Aditya Prasad, Rameswar Panda

llms benchmarks

arXiv Machine Learning

Aug 6

From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs

arXiv:2601. 03808v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches.

By Usha Shrestha, Dmitry Ignatov, Radu Timofte

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

3d ago

Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores

arXiv:2608.31068v1 Announce Type: new Abstract: When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of...

By Qiyao Yan, Chenpeng Wang, Liangming Pan

llms benchmarks

arXiv Machine Learning

Jun 16

Exploring Extrinsic and Intrinsic Properties for Effective Reasoning with Code Interpreter

arXiv:2606. 16934v1 Announce Type: cross Abstract: Reasoning with a Code Interpreter (CI) has emerged as an effective paradigm for enhancing the reasoning capabilities of large language models (LLMs) through executable computation and iterative verification.

By Patomporn Payoungkhamdee, Napat Laosaengpha, Jenta Wonglertsakul, Pittawat Taveekitworachai, Pume Tuchinda, Panjapong Poobanchuen, Ekapol Chuangsuwanich, Can Udomcharoenchaikit, Samuel Cahyawijaya, Peerat Limkonchotiwat, Sarana Nutanong

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 17

Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

arXiv:2607. 14895v1 Announce Type: new Abstract: Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding.

By Yu-Du Feng, Niels M\"undler-Sasahara, Mark Vero, Martin Vechev

llms nlp reinforcement-learning fine-tuning

arXiv AI

Jun 16

Is Code Better Than Language for Algorithmic Reasoning

arXiv:2606. 15589v1 Announce Type: cross Abstract: For tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism.

By Terry Tong, Yu Feng, Surbhi Goel, Dan Roth

llms benchmarks

arXiv AI

Jun 6

Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation

arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.

By Le Chen, Nuo Xu, Winson Chen, Bin Lei, Pei-Hung Lin, Dunzhi Zhou, Rajeev Thakur, Caiwen Ding, Ali Jannesari, Chunhua Liao

llms fine-tuning

arXiv AI

Jun 30

Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action

arXiv:2506. 13932v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks.

By Saurabh Pujar, Ira Ceka, Irene Manotas, Gail Kaiser, Baishakhi Ray, Shyam Ramji

llms agents benchmarks

arXiv AI

Jul 24

Learn2Zinc: Fine-tuning Small Language Models for Text-to-Model Translation in MiniZinc

arXiv:2607. 20456v1 Announce Type: cross Abstract: Large language models excel at code generation for mainstream programming languages but struggle with rare, domain-specific languages such as MiniZinc, a constraint modeling language for combinatorial problems.

By Serdar Kadioglu, Karthik Uppuluri

llms fine-tuning

arXiv AI

Jul 17

Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring

arXiv:2512. 14332v2 Announce Type: replace-cross Abstract: The field of Language Reasoning Models (LRMs) has been very active over the past few years with advances in training and inference techniques enabling LRMs to reason longer, and more accurately.

By Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo, John D. Kelleher

benchmarks

Related stories