Mistral AI

Rails testing on autopilot: Building an agent that writes what developers won't

arXiv AI
Jun 8

Measuring Agents in Production

arXiv:2512. 04123v4 Announce Type: replace-cross Abstract: LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful.

By Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Quincy Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, Koushik Sen, Dawn Song, Joseph E. Gonzalez, Ion Stoica, Matei Zaharia, Marquita Ellis
arXiv Computation and Language
Sep 23

WatchPoint: Executable User Feedback for Real-World Agentic Web Development

WatchPoint is a simulated‑user system that generates and runs diagnostic scripts against a live web application, producing structured observations to guide coding model retries. Unlike prior methods that rely on screenshots or non‑executable metrics, WatchPoint operates on Web‑Bench—a benchmark of 50 multi‑file web projects with 1,000 sequential tasks verified by deterministic end‑to‑end tests. It recovers 57.6% of diagnosed tasks, matching a human tester’s 54.5% recovery rate, and identifies when such simulated feedback is beneficial or should be withheld.

By Guanqun Yang, Wei Yang, Xueqing Liu
arXiv AI
Oct 15

Game-Creation-Agents: Autonomous Game Production

arXiv:2610.16776v1 Announce Type: new Abstract: Game-Creation-Agents (GCA) turns a natural-language request into a runnable Phaser 4 browser game. The pipeline first builds a multimodal game design d...

By Tencent Multimodal RL Team, Xiangxin Zhou, Jiaqi Tang, Ke Ma, Wenhao Yang, Jiarui Yao, Boye Niu, Haiyang Guo, Yifei Zhao, Huanjin Yao, Pu Jian, Ruihuang Li, Haitao Wu, Yushi Huang, Xiangyan Liu, Yan Sun, Kaixin Li, Tianyu Pang
arXiv AI
Sep 11

Introducing Consort: A Spec-First Agent Framework for Enforced, Test-Driven Development on Live Database Branches

Consort is a spec‑first, test‑driven agent framework that enforces engineering discipline through immutable controls, a deterministic orchestrator, and human‑approved gates. It operates on live database branches, guiding role agents through a spec‑first design lane and a test‑driven build lane. The framework claims that enforcing tests and gates in code keeps agent‑written code honest and verifiable, while specialized roles make it maintainable.

By Kevin Hartman
arXiv AI
Oct 12

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.

By Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu
arXiv AI
Sep 24

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

CART (Closed‑Loop Adaptive Red Teaming) is a framework that iteratively uses results from red‑teaming tests to guide subsequent probes, thereby expanding risk coverage and maintaining diversity. It separates the roles of Challenger (test generator), Target (model or agent under test), and Judge (result evaluator), enabling independent study of each component. Across multiple evaluation families, CART uncovers more failures and higher risk than static prompt replay, demonstrating that adaptive, continuous testing reveals weaknesses that fixed‑prompt methods miss.

By Dongdong Zhang, Tengchao Lv, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Furu Wei