arXiv AI
Aug 3
Agentic Harness for Real-World Compilers
arXiv:2603. 20075v2 Announce Type: replace-cross Abstract: Compilers are critical to modern computing, yet fixing compiler bugs is difficult.
LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.
Related stories
arXiv AI
Aug 5
PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
arXiv:2607. 28587v2 Announce Type: replace-cross Abstract: SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability.
arXiv AI
Sep 24
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
arXiv AI
Sep 18
BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
BuildBench introduces a realistic benchmark for evaluating large language model agents on the task of compiling open‑source software (OSS). It includes diverse OSS projects that lack clear build instructions, have undocumented dependencies, and may require source patching or script modification. The authors also present OSS‑BUILD‑AGENT, a baseline LLM‑based agent that retrieves build instructions effectively and achieves state‑of‑the‑art performance on the benchmark.
arXiv AI
Jul 2
SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests
arXiv:2607. 00990v1 Announce Type: cross Abstract: Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories.
arXiv Computation and Language
Sep 25
Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
The paper investigates how large language models (LLMs) handle bug fixing versus problem solving in competitive programming. Using a dataset of ~3,000 Codeforces submissions and their human fixes, the authors compare LLM-generated patches to human patches and assess whether LLMs prefer to modify buggy code or generate new solutions. Results show that LLMs often alter more lines than necessary and sometimes produce entirely new solutions, performing better when allowed to generate solutions from scratch rather than patching existing code.
arXiv AI
Jul 3
Understanding Agent-Based Patching of Compiler Missed Optimizations
arXiv:2607. 02370v1 Announce Type: cross Abstract: Compiler missed optimizations refer to cases in which compilers failed to optimize certain code.
arXiv AI
Jun 6
Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation
arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.
arXiv AI
Aug 28
Exploring the Role of LLMs in HPC Programming: A Survey
The survey reviews how Large Language Models (LLMs) are being used in High‑Performance Computing (HPC) programming, covering code generation, parallelization, frameworks, evaluation, and broader challenges. It finds that general‑purpose LLMs perform adequately on serial and OpenMP‑style tasks but struggle with distributed MPI workloads, while domain‑specialized models achieve higher accuracy yet are limited in scope and evaluation. The authors argue that LLMs will not replace HPC experts soon but can act as powerful collaborators, provided richer datasets, integration with performance tools, rigorous evaluation, and governance are developed.
Hugging Face Trending Papers
Sep 24
Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
The paper investigates how Large Language Models (LLMs) handle bug fixing compared to human-written patches by analyzing about 3,000 Codeforces submissions. It finds that LLMs often modify more lines than necessary and sometimes produce entirely new solutions, and that they solve more problems correctly when generating solutions from scratch rather than patching existing code. The study highlights implications for AI‑assisted programming tools, suggesting a shift toward incremental problem‑solving strategies.
arXiv AI
Jun 9
RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
arXiv:2606. 08976v1 Announce Type: new Abstract: LLM-based RTL generation and reasoning is a promising direction for hardware design automation.
arXiv AI
Jun 30
Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking
arXiv:2606. 29088v1 Announce Type: cross Abstract: There are various benchmarks to evaluate bugfixing capabilities of Large Language Models.