arXiv AI
Interaction-as-a-Judge
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 23
WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective
arXiv:2609.15387v3 Announce Type: replace-cross Abstract: Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automat...
arXiv AI
Sep 15
IWC-Bench: Evaluating Web Application Generation from a Software Testing Perspective
arXiv:2609.15387v1 Announce Type: cross Abstract: Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evalu...
arXiv AI
Oct 9
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
AgentHorizon is a benchmark comprising 1,373 computer‑use tasks with instruction‑trajectory pairs collected from 166 hours of human‑recorded activity across three operating systems. The benchmark uses a paired design that swaps instructions to create negative tasks, testing judges’ ability to distinguish successful trajectories from those that completed a similar but incompatible request. Eleven judges were evaluated, with GPT‑5.5 achieving the highest balanced accuracy of 80.9% on the full trajectory subset, while tool‑use benefits some models but harms others, revealing large variability in judges’ performance.
arXiv AI
Sep 21
GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
arXiv:2609.21293v1 Announce Type: new Abstract: Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessari...
arXiv Machine Learning
Jun 30
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
arXiv AI
Sep 1
The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents
arXiv:2608.28795v1 Announce Type: cross Abstract: Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screensho...