QTestBench is a new framework that evaluates quantum code tests generated by large language models (LLMs) by comparing their verdicts against independent benchmark checkers, rather than treating the tests as ground truth. It prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and assesses test quality across three validation cohorts. In benchmarks on PennyLane QHack challenges and Qiskit HumanEval tasks, generated test suites frequently execute but often fail to discriminate correct from faulty implementations, with the best test quality scores of 0.71 and 0.34 respectively.
By Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique
SWE-Critix is a post‑training pipeline that creates generalist verifiers for coding‑agent trajectories. It uses supervised fine‑tuning followed by reinforcement learning to train 30B models on CodeForge‑Preview data, enabling the verifier to predict completion labels and generate natural‑language explanations. The approach raises classification accuracy from 57% to 78% on a held‑out test set.
By Jiajun Hu, Chun Yong Chong
arXiv:2606. 15589v1 Announce Type: cross Abstract: For tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism.
By Terry Tong, Yu Feng, Surbhi Goel, Dan Roth
arXiv:2505. 03818v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics.
By Antonio Valerio Miceli-Barone, Vaishak Belle, Ali Payani
As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or h...
The paper introduces Req2Test-Bench, a benchmark of 5,086 instances linking natural‑language Gherkin requirements to developer‑written test implementations across five languages and four BDD frameworks. It evaluates eight agent–model–reasoning setups, finding that while the best configuration (Codex with DeepSeek‑V4.1‑Flash) produces passing tests for 82.8% of cases, only 33.8% are behaviorally adequate, as measured by kill‑completeness. The study highlights diminishing returns from scaling agents, the impact of under‑specified requirements, and the potential of requirement‑grounded harnesses to improve test generation.
By Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong