SA-Bench is a diagnostic benchmark that evaluates how well large language model (LLM) agents reproduce scientific papers by measuring semantic alignment between generated code and the papers’ specifications. It covers 30 recent machine‑learning papers, decomposing each into 1,491 Semantic Alignment Units (SAUs) that assess numerical, methodological, protocol, and ordering fidelity. Across 12 generator configurations, the best model (Claude+PaperCoder) scores only 0.301 on average, highlighting that current LLMs often implement requirements incorrectly or with stubs.
arXiv:2607. 28587v2 Announce Type: replace-cross Abstract: SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability.
PaperCompiler is a framework that translates research papers into repository-level code by compiling paper-grounded evidence into explicit implementation specifications. It preserves source provenance, distinguishes between paper-supported, inferred, externally delegated, and unresolved information, and encodes requirements such as non-degradation, ownership, cross-file dependencies, and file-level constraints. The framework improves reference-based fidelity by 13.8% and cuts high-severity evaluator critiques from 13.2% to 6.1% on Paper2CodeBench.
arXiv:2607. 02615v1 Announce Type: cross Abstract: Generating structured artifacts with Large Language Models - e.
arXiv:2607. 08981v1 Announce Type: cross Abstract: LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed.
PaperCompiler is a framework that translates research papers into repository-level code by compiling paper-grounded evidence into explicit implementation specifications. It preserves source provenance, distinguishes between paper-supported, inferred, externally delegated, and unresolved information, and encodes non-degradation requirements, ownership, cross-file dependencies, and file-level constraints. The approach outperforms baselines on Paper2CodeBench, improving reference-based fidelity by 13.8% and halving high-severity evaluator critiques.
arXiv:2606. 18237v1 Announce Type: cross Abstract: Reproducing research results from papers and released code is central to scientific progress.
arXiv:2608. 03983v1 Announce Type: cross Abstract: Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation.
arXiv:2607. 02615v2 Announce Type: replace-cross Abstract: Generating structured artifacts with Large Language Models - e.
arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.
The survey reviews how Large Language Models (LLMs) are being used in High‑Performance Computing (HPC) programming, covering code generation, parallelization, frameworks, evaluation, and broader challenges. It finds that general‑purpose LLMs perform adequately on serial and OpenMP‑style tasks but struggle with distributed MPI workloads, while domain‑specialized models achieve higher accuracy yet are limited in scope and evaluation. The authors argue that LLMs will not replace HPC experts soon but can act as powerful collaborators, provided richer datasets, integration with performance tools, rigorous evaluation, and governance are developed.
ReproAgent is a four‑stage pipeline—Prepare, Plan, Generate, Repair—that uses a persistent implementation contract to guide scientific AI agents in converting research papers into executable code repositories. The system employs two channels: an implementation‑requirement channel that translates paper snippets into code obligations, and a reference‑evidence channel that pulls content and structure from related repositories. Evaluated on PaperBench Code‑Dev, ReproAgent achieves the highest mean score among same‑backbone scaffolds for both Claude‑Sonnet‑4.5 and Gemini‑3‑Flash, with ablation studies confirming the contribution of both channels.
arXiv:2605. 15229v3 Announce Type: replace-cross Abstract: Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue.