SA-Bench is a diagnostic benchmark that evaluates how well large language model (LLM) agents reproduce scientific papers by measuring semantic alignment between generated code and the papers’ specifications. It covers 30 recent machine‑learning papers, decomposing each into 1,491 Semantic Alignment Units (SAUs) that assess numerical, methodological, protocol, and ordering fidelity. Across 12 generator configurations, the best model (Claude+PaperCoder) scores only 0.301 on average, highlighting that current LLMs often implement requirements incorrectly or with stubs.
arXiv:2607. 28587v2 Announce Type: replace-cross Abstract: SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability.
PaperCompiler is a framework that translates research papers into repository-level code by compiling paper-grounded evidence into explicit implementation specifications. It preserves source provenance, distinguishes between paper-supported, inferred, externally delegated, and unresolved information, and encodes requirements such as non-degradation, ownership, cross-file dependencies, and file-level constraints. The framework improves reference-based fidelity by 13.8% and cuts high-severity evaluator critiques from 13.2% to 6.1% on Paper2CodeBench.
arXiv:2607. 02615v1 Announce Type: cross Abstract: Generating structured artifacts with Large Language Models - e.
arXiv:2607. 08981v1 Announce Type: cross Abstract: LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed.
PaperCompiler is a framework that translates research papers into repository-level code by compiling paper-grounded evidence into explicit implementation specifications. It preserves source provenance, distinguishes between paper-supported, inferred, externally delegated, and unresolved information, and encodes non-degradation requirements, ownership, cross-file dependencies, and file-level constraints. The approach outperforms baselines on Paper2CodeBench, improving reference-based fidelity by 13.8% and halving high-severity evaluator critiques.
arXiv:2606. 18237v1 Announce Type: cross Abstract: Reproducing research results from papers and released code is central to scientific progress.
arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...
arXiv:2608. 03983v1 Announce Type: cross Abstract: Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation.
arXiv:2605.17558v2 Announce Type: replace-cross Abstract: Training tool-calling agents requires large-scale trajectory data with verifiable labels, yet existing approaches either synthesize environme...
arXiv:2607. 02615v2 Announce Type: replace-cross Abstract: Generating structured artifacts with Large Language Models - e.
Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning introduces VeriSpec, a method that audits natural‑language model specifications directly using an LLM as a verifier. VeriSpec extracts structured rules, builds a topic‑guided graph to cluster related rules, and applies LLM‑based reasoning to find inconsistencies. In a case study on the OpenAI Model Spec, it identified 405 rules, validated five inconsistencies, and outperformed five baselines with the highest precision and lowest cost per validated inconsistency.
arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.