arXiv AI

Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise

The paper discusses how enterprises increasingly deploy AI coding agent harnesses, often purchased from vendors like Anthropic or OpenAI, and how these harnesses dictate model choice, prompt handling, and cost. It introduces a fast, customizable routing system that classifies prompts and strategically routes them to minimize expensive model usage, achieving 14–21% cost savings in a simulated 10,000-seat enterprise. The study also evaluates risks across twenty harnesses, highlights vendor dependence, and proposes an internal control plane for future harness ownership decisions.

arXiv AI
Jul 9

The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI

arXiv:2607. 06906v1 Announce Type: new Abstract: Agentic AI development today runs on token maxing: buying capability with tokens -- longer reasoning traces, more turns, wider tool payloads, bigger replayed contexts -- so tokens per task grow faster than task value.

By Muayad Sayed Ali, Aliaksandra Novik, Anji Boddupally, Artem Yavorskyi, Chris Nickerson, Daniel Rica, Emily DuGranrut, Felix Leung, Garrett Prince, Grace Barnett, Heath Robinson, Hosain Al Ahmad, Jesse Resnick, Juan Carlos Farah, Jyothi Swaroop Meruga, Leonid Kuznetsov, Luke Gorham, Marie Schmoll, Michael Paciullo, Saumya Das, Sharath Sheripally, Tommy Griscom, Mykyta Osadchyi, Neha Mantri, Nick Westrum, Olivia Benowitz, Parikshith Kulkarni, Radik Chernyshov, Rakshith Vasudev, Rohith Nadimpally, Vikas Gangadevi, Waseem AlShikh
arXiv AI
Oct 16

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

JevAgentHarness is a new framework that benchmarks, deploys, and trains decision models to share control with large language models (LLMs) in agentic tasks. It introduces AgentDecisionBench to evaluate decision models from isolated questions to full agent control, revealing that accurate isolated decisions still miss many tasks and that naïve delegation can reduce success. By fitting confidence thresholds per choice and training a 4B decision model on states after agent mistakes, JevAgentHarness achieves high task success while cutting cost and speeding up execution, with results and resources made publicly available.

By Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji
arXiv AI
Nov 6

Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup

The paper reports on Grayhat Developers’ experience deploying a hybrid AI stack that combines local open‑source models with commercial agents. After finding local models too slow for complex coding tasks, the team kept commercial agents for those tasks while using local models for lower‑complexity work, documentation sync, and tool coordination, reducing commercial AI usage by 27–36%. The initial self‑hosted deployment suffered from UX friction, but a centrally configured deployment improved adoption.

By Saad Ahmed Bazaz, Talha Amjad, Abdurrehman Subhani, Sohaib Ahmed Bazaz, Abdullah Aamir
arXiv AI
Aug 24

Applying Anthropic Primitives at Large Enterprises: Harness Paradigm for Knowledge Work

The paper discusses how large enterprises can adopt the harness paradigm to overcome limitations of traditional coding approaches. It reviews recent findings that harnesses outperform complex architectures at the task level, that harness choice drives benchmark variance more than model choice, and that governance is the main barrier to enterprise adoption. The authors propose a unified harness architecture that keeps code identical across deployments, simplifying review and audit processes.

By George Juraj Salapa
arXiv AI
Oct 9

StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents

StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.

By Daksh Raghuvanshi, Ved Vedere, Yifan Wang
arXiv Computation and Language
Sep 4

Where Does Harness-Optimization Value Live? Localized Gains and the Budget-Splitting Trap in Self-Evolving LLM Agents

The paper introduces HARNESSEVO, a method that decomposes a large language model’s harness into four independently evolvable components—role, task‑strategy, tool/format‑rules, and reflection/control. Experiments on ALFWorld show that overall success rates are similar to flat‑string evolution, but the reflection/control component alone accounts for most of the performance gains. The study also finds that evenly distributing optimization budget across all slots can be detrimental; concentrating resources on the high‑credit control slot recovers lost performance, while on WebShop all slots remain ineffective, suggesting task‑specific differences in harness value.

By Michael Nguyen, Wei Chen Tan, Nurul Aisyah Hassan, Arvind Raman, Li Hua Lim, Ahmad Faiz Razak
arXiv AI
Sep 7

$\tau^\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction

The paper introduces $ au^ au$-Bench, a benchmark that turns the construction of AI agents into a measurable task. In this environment a developer agent receives real business records, client requirements, a production API, an existing codebase, and constraints on cost and models, and must deliver a complete customer‑service agent. The benchmark evaluates performance by deploying the agent against simulated users, revealing that current state‑of‑the‑art models achieve only 23.9% success while an expert‑written reference scores 82.2%.

By Quan Shi, Keshav Dhandhania, Karthik Narasimhan, Victor Barres