arXiv Machine Learning August 31, 2026 By Yuangang Li, Justin Tian Jin Chen, Ethan Yu, David Hong, Iftekhar Ahmed

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

Read the original on arXiv Machine Learning →

The paper introduces CodeRQ-Bench, the first benchmark for assessing large language model reasoning quality across coding tasks such as generation, summarization, and classification. It analyzes over a thousand mismatches from existing evaluators, identifies recurring limitations, and derives design insights that lead to a new two‑stage evaluator, VERA. Experiments show VERA outperforms strong baselines, improving AUCROC by up to 0.26 and AUPRC by up to 0.21 on four datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Oct 12

SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories

SWE-Critix is a post‑training pipeline that creates generalist verifiers for coding‑agent trajectories. It uses supervised fine‑tuning followed by reinforcement learning to train 30B models on CodeForge‑Preview data, enabling the verifier to predict completion labels and generate natural‑language explanations. The approach raises classification accuracy from 57% to 78% on a held‑out test set.

By Jiajun Hu, Chun Yong Chong

llms agents reinforcement-learning fine-tuning

arXiv Computation and Language

Sep 30

Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch

arXiv:2604.15490v2 Announce Type: replace Abstract: Recent developments in reasoning capabilities have enabled large language models to solve increasingly complex mathematical, symbolic, and logical...

By Eleanor M. Lin, David Jurgens

llms fine-tuning

arXiv Machine Learning

Jun 16

Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification

arXiv:2601. 22642v2 Announce Type: replace Abstract: Large Language Models (LLMs) show remarkable capabilities, yet their stochastic next-token prediction creates logical inconsistencies and reward hacking that formal symbolic systems avoid.

By Chuxue Cao, Jinluan Yang, Haoran Li, Kunhao Pan, Zijian Zhao, Zhengyu Chen, Yuchen Tian, Lijun Wu, Conghui He, Sirui Han, Yike Guo

llms fine-tuning benchmarks

arXiv AI

Aug 3

RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment

arXiv:2507. 22580v2 Announce Type: replace-cross Abstract: Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention.

By Marcos Fuster-Pena, David de-Fitero-Dominguez, Antonio Garcia-Cabot, Eva Garcia-Lopez

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 10

CodeAlchemy: Synthetic Code Rewriting at Scale

arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.

By Ankit Gupta, Aditya Prasad, Rameswar Panda

llms benchmarks

arXiv Machine Learning

Aug 6

From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs

arXiv:2601. 03808v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches.

By Usha Shrestha, Dmitry Ignatov, Radu Timofte

llms reinforcement-learning fine-tuning benchmarks safety

One email a morning, machine-written

Related stories

SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories

Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch

Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification

RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment

CodeAlchemy: Synthetic Code Rewriting at Scale

From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs