Upgrading agentic coding capabilities with the new Devstral models

Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

arXiv:2606. 17799v1 Announce Type: cross Abstract: Coding agents have become a major mode of software engineering, but the benchmarks we use to compare them were designed in a pre-agent era: they collapse model, harness, and environment into a single end-to-end score, typically computed against one reference solution, with no component-level signal for iteration.

Note on 24th September 2026

Simon Willison reflects on his experience with coding agents, noting that while they enable impressive feats, they also complicate software engineering. He emphasizes that fully harnessing their capabilities demands exceptional discipline and deep knowledge. The article highlights the dual nature of coding agents as both powerful tools and challenging additions to development workflows.

How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

arXiv:2609.40303v1 Announce Type: new Abstract: Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnati...

New tools for building agents

Using Local Coding Agents

Using Open-Weight Models in Local Coding Harnesses as an Alternative to Claude Code and Codex Subscriptions

Can Agents Design Libraries for Agents?

The paper introduces LibraryDesignBench, a benchmark that tests how well agents can design reusable libraries from specifications without prescribed designs. It evaluates libraries by measuring the correctness and simplicity of programs written by three different user agents across 242 programming problems in four languages. Findings show that while agents often replicate human-designed abstractions, downstream agents still tend to reimplement library features due to rigidity or usability issues, and that providing more prescriptive guidance improves reuse and program simplicity.