← Back to all news
arXiv AI By Phanindra Reddy Madduru

When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor

Read the original on arXiv AI →

The paper reports a case study of a large language model (LLM) coding agent tasked with building a multi‑component data system from a detailed specification. During a single session the agent introduced five defects, which were categorized by violated constraints and detection methods. The study also evaluates the agent’s retrieval‑filtering strategy on the HotpotQA benchmark, showing that filtering to a graph‑identified entity set yields higher recall than unfiltered search, with a statistically significant gap across all tested budgets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 28

CARGO: Context-Aware Retrieval-Gated Evaluation of Agentic AI in Production

CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.

By Mukul Chhabra, Shail Patel, Luigi Medrano
llmsagentssafety
More like this →
arXiv AI
1d ago

Locating and Closing Information Gaps in Multi-Agent Systems

The paper introduces SPLICE, a method that identifies and closes information gaps in multi‑agent workflows for web‑task completion. SPLICE locates missing evidence after an initial run, adds a single directed edge to deliver the evidence, re‑executes affected agents, and uses a selector trained on process features to decide whether to adopt the new answer. Experiments on five benchmarks show SPLICE improves web‑tool use scores by 8.0% and reasoning/planning scores by 5.7% with modest cost increases and zero‑shot transfer to new backbones.

By Yaxin Gong, Chongming Gao, Gangyi Zhang, Chenxiao Fan, Leyang Shen, Jiakai Wang, Dong Wang, Yang Liu, Haoyan Liu, Xinting Hu
llmsagentsbenchmarks
More like this →
arXiv AI
Jul 3

ContextNest: Verifiable Context Governance for Autonomous AI Agent

arXiv:2607. 02116v1 Announce Type: new Abstract: Autonomous AI agents increasingly depend on external knowledge stores, yet most retrieval pipelines provide relevance without durable guarantees of provenance, version identity, integrity, traceability, or point-in-time reconstruction.

By Misha Sulpovar (PromptOwl, LLC), Benn R. Konsynski (Goizueta Business School, Emory University), Qaish Kanchwala (IBM Research), Gabe Goodhart (IBM Research)
ragagents
More like this →