← Back to all news
arXiv AI By Akash Gokul, Srijan Bansal, Yingbo Zhou, Semih Yavuz

TEX: Test-Time Scaling Testing Agents via Execution-based Cross-Validation

Read the original on arXiv AI →

TEX is a new test‑time scaling framework that uses execution‑driven cross‑validation to generate and refine tests for language‑model agents. By running agents in parallel and cross‑executing each test against every patch, TEX builds a $K imes K$ feedback matrix that guides objective refinement without sharing verbose traces. On the SWT‑Bench Verified benchmark, TEX reaches 86.4% accuracy—setting a new state‑of‑the‑art and improving over single‑agent baselines by 9.0%—and even random selections from its outputs outperform the previous best score.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.

By Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 15

SWEPatchTest: Evaluating Software Tests Beyond Reproduction

SWEPatchTest is a benchmark that evaluates software tests by measuring their ability to reproduce issues, accept correct patches, and reject incorrect ones across 619 tasks and 82,579 candidate patches. It introduces the Joint Validation Rate metric, revealing that even the best configurations only achieve 9.21% success when all three criteria are met. Using these validation scores to select demonstrations for supervised fine‑tuning can roughly triple the Joint Validation Rate and enhance downstream repair performance on multiple SWE‑bench datasets.

By Hua Lu, Pengcheng Wang, Han Li, Zichen Tian, Yanyu Chen, Huang He, Bingjin Chen
agentsfine-tuningbenchmarks
More like this →
arXiv Computation and Language
Aug 27

Scalable Supervision for Software Agents via Patch Reasoning

The paper introduces R4P, a reasoning‑based supervision method for software agents that eliminates the need for test execution by using a group‑wise training objective to verify multiple patches simultaneously. R4P achieves 72.2% accuracy on the SWE‑bench patch verification task, matching proprietary models, and enables the creation of an execution‑free scaffold called Mini‑SE. Mini‑SE, trained purely with reinforcement learning via R4P, improves Pass@1 from 26.2% to 32.8% over the baseline Qwen3‑32B, demonstrating R4P’s practical utility and scalable performance.

By Junjielong Xu, Boyin Tan, Xiaoyuan Liu, Chao Peng, Pengfei Gao, Pinjia He
llmsagentsreinforcement-learning
More like this →