arXiv:2606. 29088v1 Announce Type: cross Abstract: There are various benchmarks to evaluate bugfixing capabilities of Large Language Models.
arXiv:2607. 00990v1 Announce Type: cross Abstract: Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories.
arXiv:2606. 09956v1 Announce Type: cross Abstract: The rapid adoption of LLM-powered code generation has dramatically accelerated software development, yet effective verification methods remain severely underdeveloped.
arXiv:2606. 03852v1 Announce Type: cross Abstract: Large language models often generate code with bugs.
The paper investigates how large language models (LLMs) handle bug fixing versus problem solving in competitive programming. Using a dataset of ~3,000 Codeforces submissions and their human fixes, the authors compare LLM-generated patches to human patches and assess whether LLMs prefer to modify buggy code or generate new solutions. Results show that LLMs often alter more lines than necessary and sometimes produce entirely new solutions, performing better when allowed to generate solutions from scratch rather than patching existing code.
arXiv:2608. 12246v1 Announce Type: cross Abstract: Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases.
arXiv:2606. 23759v1 Announce Type: cross Abstract: Verilog debugging remains one of the most time-consuming stages in digital circuit design.
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions.
arXiv:2505. 07372v3 Announce Type: replace-cross Abstract: This paper presents a novel methodology for enhancing Automated Program Repair (APR) through synthetic data generation utilizing Large Language Models (LLMs).
arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.
The paper investigates how Large Language Models (LLMs) handle bug fixing compared to human-written patches by analyzing about 3,000 Codeforces submissions. It finds that LLMs often modify more lines than necessary and sometimes produce entirely new solutions, and that they solve more problems correctly when generating solutions from scratch rather than patching existing code. The study highlights implications for AI‑assisted programming tools, suggesting a shift toward incremental problem‑solving strategies.
FuzzingBrain‑Bench V1 is a new benchmark that tests large language models (LLMs) on their ability to discover software bugs in open‑source projects. Unlike prior benchmarks that focus on a single target vulnerability, this benchmark gives models a Docker‑based harness and asks them to generate inputs that trigger as many distinct crashes as possible. The first version contains 77 challenges from 43 projects (36 C, 32 C++, 9 Java/JVM) and evaluates Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8, with Claude Opus 4.8 achieving the highest score by triggering crashes in 60 of 77 challenges.
arXiv:2607. 04537v1 Announce Type: cross Abstract: Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct.