arXiv Machine Learning
Jun 10
CodeAlchemy: Synthetic Code Rewriting at Scale
arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.
SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation
arXiv:2607. 19104v1 Announce Type: cross Abstract: Large language models (LLMs) excel at general-purpose code generation, yet how well they handle scientific code remains an open question.
Related stories
arXiv AI
5d ago
CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
arXiv:2603. 14501v2 Announce Type: replace-cross Abstract: Large Language Models excel in high-resource programming languages but struggle with low-resource ones.
arXiv AI
Jun 30
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
arXiv:2511. 06090v3 Announce Type: replace-cross Abstract: Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness.
arXiv AI
Jun 4
CodegenBench: Can LLMs Write Efficient Code Across Architectures?
arXiv:2606. 04023v1 Announce Type: cross Abstract: While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.
arXiv AI
Jul 8
Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark
arXiv:2607. 05443v1 Announce Type: cross Abstract: Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging.
arXiv AI
Aug 6
SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models
arXiv:2608. 04975v1 Announce Type: cross Abstract: SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code.
arXiv AI
Jul 21
FormulaCode: Evaluating Agentic Optimization on Large Codebases
arXiv:2603. 16011v3 Announce Type: replace-cross Abstract: Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints.
arXiv AI
Jul 22
FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.
arXiv AI
Jun 2
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
arXiv:2606. 01286v1 Announce Type: cross Abstract: The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal.
arXiv Machine Learning
Jul 22
Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator
arXiv:2607. 18642v1 Announce Type: new Abstract: Mined code corpora are abundant but uncontrolled: a snippet's semantics, surface "messiness," and difficulty are whatever the wild contained; there is no known-optimal reference to grade against; and any public sample may already sit in a model's training set.
arXiv AI
Jul 7
Formal Disco: Scalable Open-Ended Generation of Formally Verified Programs
arXiv:2607. 04631v1 Announce Type: new Abstract: The cost of producing code is rapidly diminishing with increasingly capable AI agents, while quality assurance of generated programs has not kept pace.
arXiv AI
Jun 16
FasterPy: An LLM-based Code Execution Efficiency Optimization Framework
arXiv:2512. 22827v2 Announce Type: replace-cross Abstract: Code often suffers from performance bugs.