arXiv AI
CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 25
SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback
SciWalker is a framework that automatically synthesizes scientific coding problems by sampling operator chains from scientific library interfaces and using execution feedback to refine generated problem statements, solutions, and tests. It produces 8,178 high‑quality problems across five scientific domains and 32 subdomains, and training a large language model with these problems improves its scientific coding accuracy by nearly 10 percentage points. The approach combines structured workflow composition with verification and quality review to enable scalable, scientifically grounded task generation.
arXiv AI
Jun 2
BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
arXiv:2606. 01286v1 Announce Type: cross Abstract: The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal.
arXiv Machine Learning
Aug 31
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
The paper introduces CodeRQ-Bench, the first benchmark for assessing large language model reasoning quality across coding tasks such as generation, summarization, and classification. It analyzes over a thousand mismatches from existing evaluators, identifies recurring limitations, and derives design insights that lead to a new two‑stage evaluator, VERA. Experiments show VERA outperforms strong baselines, improving AUCROC by up to 0.26 and AUPRC by up to 0.21 on four datasets.
arXiv AI
Oct 1
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
arXiv Computation and Language
Oct 16
WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents
WEAVE is a framework that converts verified pull‑request histories into executable, dependency‑structured tasks for code agents. It organizes related PRs into bounded directed acyclic graphs (DAGs) using repository‑calibrated typed change loci, creating unified tasks that reflect real‑world code evolution. The framework also introduces Graph Credit, a deterministic, test‑grounded metric that measures partial progress consistent with dependencies, providing dense, verifiable rewards for reinforcement learning.
arXiv Machine Learning
Jun 10
CodeAlchemy: Synthetic Code Rewriting at Scale
arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.