arXiv Computation and Language

Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite

arXiv AI
6d ago

ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases

ExpertCrowd‑SWE (EC‑SWE) is a benchmark comprising 100 expert‑authored, multi‑file software engineering tasks designed to evaluate autonomous coding agents on large, unfamiliar codebases. Each task is a real change request from a practising engineer, paired with a sealed reference solution, a task‑specific grader that awards partial credit, and must‑pass blockers that mimic a maintainer’s merge criteria. The benchmark covers 87 open‑source projects across 11 languages, and the final 100 tasks were selected for their difficulty against current frontier models, with seven models evaluated over 3,495 runs.

By Josip Vrdoljak, Akash Singh, Bas Steunebrink, Alex Yarotsky, John Lorenzo, Achille Nicoletti, Sam Pigott
arXiv AI
Oct 2

Finding the Right Fit: Model-Harness Interactions across Agent Tasks

The study evaluates 66 combinations of four configurable harnesses and five language models across three benchmark suites, revealing that model rankings and best harnesses vary significantly with the task. Notably, the openJiuwen harness consistently yields the highest scores for the Kimi model, while GPT performs best with the lean PI scaffold on Terminal‑Bench 4. The findings demonstrate that a model’s performance is highly dependent on the harness it is paired with, and that higher cost or a vendor’s own harness does not guarantee superior results.

By Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng, Zhiyi Lyu, Xuyu Dong, Feng Chen, Bo An
arXiv AI
Nov 6

The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks

The paper introduces agent‑native primitives—reusable building blocks that embed a language model within a resource to apply it contextually—and HEELO, a harness‑engineering framework that composes these primitives into task‑specific graphs, verifies outcomes, and iteratively refines or adds primitives. Using eight primitive families, the authors demonstrate significant performance gains across multiple benchmarks, raising Terminal‑Bench 4.0 resolution from 37.3% to 55.5% and improving pass rates on ToolBench and GAIA‑Text‑103. When integrated with HEELO, Claude Opus 5.5 and GPT‑6.1 Sol achieve higher completion counts, faster runtimes, and lower costs on 46 real‑world tasks.

By Haibo Jin, Peng Kuang, Xiaopeng Yuan, Lucheng Fu, Jerry Wang, Xucheng Yu, Yaoning Yu, Dehao Wu, Runguo Li, Colin Su, Jingxuan Sun, Xuyang Liu, Haojing Luo, Haohan Wang
Hugging Face Trending Papers
Nov 5

The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks

The paper introduces agent‑native primitives—reusable building blocks that embed a resident LLM into resources, enabling them to apply themselves in context and communicate with other primitives. It presents HEELO, a harness‑engineering framework that composes these primitives into task‑specific graphs, verifies outcomes, refines failures, and incorporates new primitives. Experiments show significant performance gains across multiple benchmarks, with HEELO‑enhanced agents completing more real‑world tasks faster and at lower cost.

arXiv AI
Aug 28

Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance

The paper evaluates a manager‑worker scaffold that uses a shared filesystem workspace to orchestrate multi‑agent large language model (LLM) coding tasks without training or tuning. Across nine models—including five open‑weight and four closed‑weight systems—the scaffold yields statistically significant accuracy gains for some models (e.g., Qwen3.8‑27B, GPT‑5.6‑Luna, GPT‑5.6‑Terra, Kimi‑K3, Minimax‑M3) while producing null or negative effects for others (e.g., Qwen3.6‑35B). The study shows that the manager can triple token usage but still achieves higher accuracy at a fraction of the cost compared to larger single‑pass models, with key mechanisms identified as context management and problem decomposition.

By Victor Gao (Sang Won), Vida Khosrowshahi (Sang Won), Ali Khosrowshahi (Sang Won), Xihao Sun (Sang Won), Juhyun Lee (Sang Won), Simon (Sang Won), Lee
arXiv AI
Aug 25

Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep

The study evaluates how to best split tasks among large‑language‑model agents for cross‑border VAT determination, comparing one broad agent to configurations ranging from one to five narrow agents. Across 4,400 runs—including token‑matched and failure‑injection scenarios—the intermediate configurations achieved the highest accuracy but did not surpass the fine‑endpoint benchmark, leaving the optimal decomposition hypothesis unconfirmed. The pilot provides a preregistered heuristic for right‑sizing decomposition, along with an oracle, dataset, and analysis pipeline.

By Pedro Santos
arXiv Computation and Language
Oct 12

SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/​Codex CLI as the top solver and Claude Opus 4.8/​mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.

By Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo
arXiv AI
Oct 28

kw-bench: Breaking Frontier Harnesses on Real-World Tasks

The paper introduces kw-bench, a benchmark that evaluates coding-agent harnesses on investigative knowledge work, specifically a five‑year VAT reconstruction task across multiple SAP systems. It contrasts generative benchmarks with this real‑world, evidence‑heavy task, showing that model‑agnostic harnesses performed well while lab harnesses failed. The study demonstrates how frontier models can dramatically reduce time and cost for complex audit‑style engagements.

By George Juraj Salapa
arXiv Machine Learning
Sep 15

GRADE: Graph Representation of LLM Agent Dependency and Execution

The paper introduces GRADE, a graph-based representation of large language model (LLM) agent executions that captures both execution steps and their dependencies. By adding graded dependency edges—observed, declared, or inferred—to the trace, the authors evaluate how this dependency layer affects failure prediction across six corpora involving tool use, coding, and web tasks. Experiments show that the dependency block can improve prediction in some settings, but its effectiveness varies with the evaluation probe and corpus, and controlled experiments demonstrate that the observed structure is not merely a degree-matched artifact.

By Yue Zhao
arXiv AI
Sep 7

What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents

The study investigates how multi‑harness reinforcement learning (RL) affects coding agents by comparing two grouping strategies—Within (one group per task‑harness pair) and Cross (harnesses pooled within a task)—using a Qwen3‑8B policy trained on frozen task‑harness records from Aider, OpenHands, Qwen Code, and SWE‑agent. Across 24,000 sealed evaluations, the choice of evaluation harness dramatically increases solve rates (from 2.14 % to 9.27 %), while the grouping rule has a negligible effect. Both grouping rules yield similar gains on the same source harness, and Cross‑harness credit does not improve portability beyond Within‑harness credit, suggesting that multi‑harness RL reports should specify grouping boundaries and test on unseen harnesses.

By Chenqian Le, Jiayi Cheng, Qijia He, Runhao Li, Yinghao Li, Xupeng Chen