arXiv Machine Learning

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

QTestBench is a new framework that evaluates quantum code tests generated by large language models (LLMs) by comparing their verdicts against independent benchmark checkers, rather than treating the tests as ground truth. It prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and assesses test quality across three validation cohorts. In benchmarks on PennyLane QHack challenges and Qiskit HumanEval tasks, generated test suites frequently execute but often fail to discriminate correct from faulty implementations, with the best test quality scores of 0.71 and 0.34 respectively.

arXiv Machine Learning
Sep 18

QEncodeBench: Can Large Language Models Encode Classical Problems into Verified Quantum Oracles?

QEncodeBench evaluates whether large language models can translate classical constraint problems into verified quantum phase oracles. The benchmark measures the correctness of generated circuits using an adversarial self‑validated verifier that checks full solution‑set equivalence while enforcing resource limits. Results show that models lacking a reasoning mode perform poorly, whereas enabling native reasoning improves accuracy tenfold; semantic errors dominate, and neuro‑symbolic pipelines close most gaps by delegating critical composition to deterministic procedures.

By Xujun Che, Hanhan Wu, Yuchen Yuan, Chenyang Yu
Hugging Face Trending Papers
Sep 28

QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

QC-Stark is a benchmark that evaluates large language models on 11 quantum computing tasks, including circuit construction, debugging, compilation, error correction, and simulation. It comprises 2,750 evaluations across 10 models, 5 difficulty levels, and 5 seeds, revealing that overall model rankings can hide significant per-task differences, with Spearman correlation being insignificant for 4 of the 11 tasks. The benchmark’s measurement quality is validated by a 2‑parameter Item Response Theory model, prompt sensitivity analysis confirms ranking robustness, and all tasks are auto‑verifiable via execution, eliminating the need for manual evaluation. The code and data are publicly available on Hugging Face.

arXiv AI
Jul 24

PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generation

arXiv:2605. 25572v2 Announce Type: replace-cross Abstract: The growing complexity of quantum programming frameworks has exposed a critical limitation in existing large language model (LLM)-based code assistants: general-purpose models hallucinate PennyLane-specific gate names, misplace device configurations, and produce structurally invalid circuits when faced with specialized quantum coding challenges.

By Minghao Shao, Nouhaila Innan, Hariharan Janardhanan, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique
arXiv AI
Jul 13

QAgent: An LLM-based Multi-Agent System for Autonomous OpenQASM programming

arXiv:2508. 20134v2 Announce Type: replace Abstract: Programming quantum circuits at the OpenQASM level is essential for achieving hardware-aware optimization and reliable execution on noisy intermediate-scale quantum (NISQ) devices, yet it remains challenging due to the need for domain-specific planning, iterative code synthesis, and low-level calibration.

By Zhenxiao Fu, Lei Jiang, Yilun Xu, Gang Huang, Fan Chen
arXiv AI
Aug 11

QuantumMind: Constraint-Grounded Agentic Reasoning for Speedup Analysis in Quantum Computing

arXiv:2608. 07743v1 Announce Type: new Abstract: Identifying a meaningful quantum speedup requires more than matching a classical problem to a familiar quantum primitive: the claim must preserve the task, respect access and output models, expose required promises, and remain within a defensible complexity scope.

By Yijing Zuo, Zhe Fu, Zihan Nie, Zhihui Zhu, Haohan Wang
arXiv AI
Sep 16

QART: A Quantum-Classical Hybrid Architecture for Long-Horizon Reasoning -- Exploring a Conditional Path toward Quantum Scaling

QART is a quantum‑classical hybrid architecture that augments a language model with quantum encoding, CIM‑based QUBO optimization, and quantum decoding to improve long‑horizon reasoning. The authors claim that, under certain assumptions, QART can maintain a non‑zero probability of recovering an optimal reasoning path while traditional autoregressive LLMs see their acceptance probability drop to zero as cumulative risk grows. Experiments on six benchmarks with three backbone models show that QART outperforms the baselines in 14 of 15 pairings, with relative gains up to 84.0% on SciCode.

By Lehao Lin, Yuheng Cheng, Guolong Liu, Yao Li, Xuning Tan, Xiyuan Zhou, Ruixi Zou, Shi Wang, Huan Zhao, Wenxuan Liu, Haifeng Wu, Junhua Zhao
arXiv AI
Oct 19

A Theoretical Analysis of Test-Driven Code Generation

The paper presents a probabilistic framework for two main test‑driven code generation paradigms: selecting code after generation using an execution environment, and generating code conditioned on environment feedback. It formalizes selection heuristics as environment‑aware estimators of code correctness, proving that fuzzy functional similarity estimators provide a stronger inductive bias and higher signal‑to‑noise ratio than functional equivalence estimators. Additionally, it models backprompting as an in‑context approximation of Thompson sampling, deriving a regret bound that explains its limitations due to ambiguous task descriptions, and validates these theoretical insights with experiments on seven open‑weight models across multiple benchmarks, including a new quantum programming benchmark.

By Nicolas Menet, Michael Hersche, Andreas Krause, Abbas Rahimi
arXiv AI
Sep 17

BadQubits: An LLM-Based Framework for Static Pre-Execution Detection of Structurally Harmful Quantum Circuits

BadQubits is an LLM-based framework that statically analyzes OpenQASM 2.0 circuits before execution to detect structurally harmful patterns. The system evaluates four large language model architectures on 1,500 circuits, achieving 92.67% classification accuracy and 96.1% recall for harmful circuits with a fine‑tuned Qwen Coder 2.5 7B model. Comparative experiments show that LLMs retain sequential token structure, outperforming a bag‑of‑gates CNN, and that model decisions correlate with threat‑defining features such as SWAP density and measurement timing.

By Justin Woodring, Lamine Noureddine, Aisha Ali-Gombe