← Back to all news
arXiv AI By Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng

From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

Read the original on arXiv AI →

The paper introduces MCR-Bench, a benchmark for realistic multi‑round code review that includes 2,269 real‑world tasks across five programming languages, each annotated with fine‑grained defect information and dynamic state labels. Experiments with mainstream large language models show limited overall performance, especially as interaction rounds increase, and reveal that model accuracy varies by defect type and severity. Error analysis identifies key failure mechanisms such as cross‑round temporal misalignment and insufficient long‑range memory.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 16

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

The paper introduces AXIOM, a perturbation-based framework for creating a balanced benchmark of code evaluation problems. AXIOM refines program quality by applying controlled semantic perturbations and then calibrates scores through selective manual annotation, resulting in the multilingual AXIOMBENCH dataset with 1,962 problems. This approach addresses limitations of prior benchmarks that rely on coarse binary labels or subjective human scoring.

By Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao
llmsbenchmarkssafety
More like this →