← Back to all news
arXiv Computation and Language By Huifei Wang, Xinying Huang, Yiheng Sun, Yifan Yuan

ReMCTS: Reflection-Enhanced Monte Carlo Tree Search for Code Generation

Read the original on arXiv Computation and Language →

ReMCTS is a Monte Carlo Tree Search–style framework that enhances code generation by large language models (LLMs). It organizes program candidates as tree states, retains branch‑local debugging context, and retrieves failure experience across branches, distinguishing failed checks from unavailable evidence. Experiments on HumanEval and MBPP‑Sanitized show that visible‑test ReMCTS improves performance over direct generation in 8 of 10 model‑dataset pairs, while a pilot on HumanEval‑X C++ demonstrates compatibility with compiler‑backed execution.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Jul 14

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.

By Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, Valentin Malykh
llmsagentsbenchmarks
More like this →
arXiv AI
Aug 28

From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

The paper introduces MCR-Bench, a benchmark for realistic multi‑round code review that includes 2,269 real‑world tasks across five programming languages, each annotated with fine‑grained defect information and dynamic state labels. Experiments with mainstream large language models show limited overall performance, especially as interaction rounds increase, and reveal that model accuracy varies by defect type and severity. Error analysis identifies key failure mechanisms such as cross‑round temporal misalignment and insufficient long‑range memory.

By Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng
llmsbenchmarks
More like this →
arXiv AI
Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
llmsbenchmarks
More like this →