The paper introduces Rondo, a stateful testing framework for deep‑learning compilers that models each test as a model plus an invocation sequence. Rondo mutates sequences using a Markov chain Monte Carlo sampler guided by compilation behavior, and employs a three‑way differential oracle to detect bugs that arise only when compiler state is retained. Evaluated on four major frameworks and 24 model architectures, Rondo uncovered 117 bugs—35 of which were stateful—leading to 39 bug reports, 35 confirmed by developers, and 18 fixes, while achieving high coverage and precision.
By Jiatao Ge, Ruixiang Qian, Yibo Hu, Yanzhou Mu, Yinglong Zou, Jiawei Liu, Zhenyu Chen, Chunrong Fang
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu
The paper introduces OMVV, an online multi-verifier algorithm that maintains a pool of weak verifiers with varying costs and performance. It adaptively selects a verifier each round using an online score combiner and exponential-weights routing, providing distribution-free guarantees on false-accept and false-reject rates. Experiments on reasoning benchmarks show OMVV achieves higher accuracy at lower verification cost than any single fixed verifier across different budgets.
By Erfan Hajihashemi, Yanning Shen
The paper presents a method for probabilistic model checking of autoregressive neural sequence models. It extracts a discrete-time Markov chain from the model’s token generation, verifies PCTL specifications with PRISM, and aggregates verdicts into a coverage curve over the input space. The approach includes a soundness theorem, a counterexample-guided refinement loop, and a maximum-likelihood algorithm to find the most probable falsifying trace, demonstrated on GPT‑2 and a SMILES molecular generator.
By Helge Spieker, Dennis Gross, Arnaud Gotlieb
arXiv:2603. 25450v2 Announce Type: replace Abstract: Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment.
By Matt Gorbett, Suman Jana
The paper introduces a risk‑controlled framework for using large language models (LLMs) as judges in tasks without reference answers. By calibrating uncertainty thresholds on a held‑out set, the method ensures that the false discovery rate of accepted verdicts stays below a user‑specified level α with high probability, using finite‑sample Clopper–Pearson intervals. When the parametric judge lacks confidence, the instance is routed to a retrieval‑augmented mode with a second calibrated threshold, preserving the error guarantee while achieving higher coverage than single‑mode baselines.
arXiv:2607. 22602v1 Announce Type: new Abstract: Inference-time scaling has emerged as a powerful paradigm for improving large language model reasoning, often delivering larger gains on difficult reasoning tasks than parameter scaling alone.
By Tingxin Yang, Zefeng Wang, Mengyue Wang, Xingcheng Zhou, Yunpu Ma
SCOPE compiles natural‑language policies into neurosymbolic predicate programs that combine small neural models, deterministic extractors, and logical composition. The system is jointly optimized with adversarially generated counterexamples, achieving higher accuracy than prompt‑optimization, task‑decomposition, and workflow‑generation baselines on real‑world policy benchmarks. Programs trained with the weakest executor transfer to other models, enabling deployment on inexpensive models with stable, interpretable predictions.
By Arnon Mazza, Elad Levi
DeepLook is a training‑free decoding framework that uses segment‑level confidence to guide branching during inference. It only expands the search tree at points of high uncertainty and ranks candidate continuations by Average Lookahead Confidence (ALC), a mean confidence over fixed‑horizon rollouts, before finalizing and voting on a pruned subset. Experiments on four mathematics benchmarks show that DeepLook matches or surpasses DeepConf‑low in 12 of 16 settings while reducing token usage by 87.3%, though its advantage disappears on the knowledge‑intensive GPQA‑Diamond task due to hallucinations receiving high ALC.
By Tingxin Yang, Zefeng Wang, Mengyue Wang, Jinhe Bi, Xingcheng Zhou, Volker Tresp, Yunpu Ma
arXiv:2512. 02080v3 Announce Type: replace-cross Abstract: The integration of Formal Verification tools with Large Language Models (LLMs) offers a path to scale software verification beyond manual workflows.
By Pierre Dantas, Lucas Cordeiro, Youcheng Sun, Waldir Junior
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
By Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade
arXiv:2607. 01280v1 Announce Type: new Abstract: Programming-by-example systems infer programs from a small set of input-output examples.
By Yuan Si, Jialu Zhang