arXiv:2604. 11556v2 Announce Type: replace-cross Abstract: LLM-assisted software development has become increasingly prevalent, and can generate large-scale systems, such as compilers.
arXiv:2607. 04631v1 Announce Type: new Abstract: The cost of producing code is rapidly diminishing with increasingly capable AI agents, while quality assurance of generated programs has not kept pace.
E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
Spec‑Harness evaluates how well large language models (LLMs) synthesize Java Modeling Language (JML) specifications by measuring behavioral adequacy across precondition and postcondition correctness and completeness. The study shows that while prompt optimization can raise verifier pass rates, many accepted specifications remain behaviorally weak, either over‑ or under‑constraining inputs and outputs. Spec‑Harness also serves as a feedback mechanism that improves the quality of specifications generated by general‑purpose coding agents and a specialized JML agent.
arXiv:2506. 13932v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks.
arXiv:2606. 06523v1 Announce Type: new Abstract: Equipping Large Language Models (LLMs) to execute reliable multi-step workflows has become a central challenge in artificial intelligence.
arXiv:2604. 11950v2 Announce Type: replace-cross Abstract: While recent LLM-based agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection.
arXiv:2606. 05608v2 Announce Type: replace-cross Abstract: For over half a century, software engineering has operated on a foundational premise: human engineers decompose problems, encode decision logic into static code, and manually adapt that code as requirements evolve.
The paper introduces JAZ, a minimalist LLM agent framework that centers on a single primitive called “invoke”, which allows an LLM to write and execute arbitrary code, including recursive calls, while treating all inputs and interaction history as variables in the code environment. JAZ provides built‑in hooks for constraints and monitoring but relies solely on prompting, without external tools, memory systems, or file‑system access. Experiments show that JAZ “invoke” outperforms specialized external harnesses such as Letta (MemGPT) and ACE on long‑horizon recall tasks and continual self‑improvement, achieving higher accuracy at lower cost.
arXiv:2606. 05608v1 Announce Type: cross Abstract: For over half a century, software engineering has operated on a foundational premise: human engineers decompose problems, encode decision logic into static code, and manually adapt that code as requirements evolve.
The paper introduces ARTEMIS, a no-code evolutionary optimization platform that automatically tunes large language model (LLM) agents by jointly optimizing prompts, tool descriptions, and parameters using semantically-aware genetic operators. Starting from a benchmark script and natural language goals, ARTEMIS discovers configurable components, extracts performance signals from execution logs, and evolves configurations without architectural changes. Experiments on four agent systems show significant gains: a 13.6% increase in acceptance rate for the ALE Agent, a 10.1% performance boost for the Mini‑SWE Agent, a 36.9% token‑reduction for the CrewAI Agent, and a 22% accuracy improvement for the MathTales‑Teacher Agent using a smaller open‑source model.
arXiv:2606. 31134v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated exceptional capabilities in mathematical reasoning, they frequently produce subtle errors that evade human detection.
arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wi...