← Back to all news
arXiv Machine Learning By Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

Read the original on arXiv Machine Learning →

QTestBench is a new framework that evaluates quantum code tests generated by large language models (LLMs) by comparing their verdicts against independent benchmark checkers, rather than treating the tests as ground truth. It prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and assesses test quality across three validation cohorts. In benchmarks on PennyLane QHack challenges and Qiskit HumanEval tasks, generated test suites frequently execute but often fail to discriminate correct from faulty implementations, with the best test quality scores of 0.71 and 0.34 respectively.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 18

QEncodeBench: Can Large Language Models Encode Classical Problems into Verified Quantum Oracles?

QEncodeBench evaluates whether large language models can translate classical constraint problems into verified quantum phase oracles. The benchmark measures the correctness of generated circuits using an adversarial self‑validated verifier that checks full solution‑set equivalence while enforcing resource limits. Results show that models lacking a reasoning mode perform poorly, whereas enabling native reasoning improves accuracy tenfold; semantic errors dominate, and neuro‑symbolic pipelines close most gaps by delegating critical composition to deterministic procedures.

By Xujun Che, Hanhan Wu, Yuchen Yuan, Chenyang Yu
llmsagents
More like this →
Hugging Face Trending Papers
Sep 28

QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

QC-Stark is a benchmark that evaluates large language models on 11 quantum computing tasks, including circuit construction, debugging, compilation, error correction, and simulation. It comprises 2,750 evaluations across 10 models, 5 difficulty levels, and 5 seeds, revealing that overall model rankings can hide significant per-task differences, with Spearman correlation being insignificant for 4 of the 11 tasks. The benchmark’s measurement quality is validated by a 2‑parameter Item Response Theory model, prompt sensitivity analysis confirms ranking robustness, and all tasks are auto‑verifiable via execution, eliminating the need for manual evaluation. The code and data are publicly available on Hugging Face.

llmsbenchmarks
More like this →
arXiv AI
Jul 24

PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generation

arXiv:2605. 25572v2 Announce Type: replace-cross Abstract: The growing complexity of quantum programming frameworks has exposed a critical limitation in existing large language model (LLM)-based code assistants: general-purpose models hallucinate PennyLane-specific gate names, misplace device configurations, and produce structurally invalid circuits when faced with specialized quantum coding challenges.

By Minghao Shao, Nouhaila Innan, Hariharan Janardhanan, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique
llmsrag
More like this →