The paper introduces Rondo, a stateful testing framework for deep‑learning compilers that models each test as a model plus an invocation sequence. Rondo mutates sequences using a Markov chain Monte Carlo sampler guided by compilation behavior, and employs a three‑way differential oracle to detect bugs that arise only when compiler state is retained. Evaluated on four major frameworks and 24 model architectures, Rondo uncovered 117 bugs—35 of which were stateful—leading to 39 bug reports, 35 confirmed by developers, and 18 fixes, while achieving high coverage and precision.
arXiv:2606. 20128v1 Announce Type: cross Abstract: Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks.
arXiv:2606. 29088v1 Announce Type: cross Abstract: There are various benchmarks to evaluate bugfixing capabilities of Large Language Models.
arXiv:2606. 26492v1 Announce Type: cross Abstract: Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task.
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
arXiv:2606. 27396v1 Announce Type: cross Abstract: Test-input generation for tensor kernels is folkloric.
The paper introduces mutation analysis as a metric for evaluating GPU‑kernel benchmark oracles, injecting over ten thousand faults into verified CUDA implementations of 188 KernelBench problems. It shows that the current official checkers miss 16.9% of faults, with precision faults being especially problematic, and demonstrates that optimized test suites can achieve 98% detection with only two inputs per problem. The study also reveals flaws in existing patches and a fuzzing recipe that incorrectly rejects correct kernels 107 times.
arXiv:2606. 15994v1 Announce Type: new Abstract: Translating deep learning models from PyTorch's flexible, object-oriented design to JAX's functional, stateless setup is usually a manual and error-prone task.
FuzzingBrain‑Bench V1 is a new benchmark that tests large language models (LLMs) on their ability to discover software bugs in open‑source projects. Unlike prior benchmarks that focus on a single target vulnerability, this benchmark gives models a Docker‑based harness and asks them to generate inputs that trigger as many distinct crashes as possible. The first version contains 77 challenges from 43 projects (36 C, 32 C++, 9 Java/JVM) and evaluates Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8, with Claude Opus 4.8 achieving the highest score by triggering crashes in 60 of 77 challenges.
arXiv:2607. 12868v1 Announce Type: cross Abstract: Deep learning systems often fail due to subtle implementation faults that alter training behavior.
arXiv:2606. 04310v1 Announce Type: new Abstract: Deep Neural Networks (DNNs) are increasingly being deployed in security-critical and safety-sensitive applications, which makes rigorous testing essential to identify and mitigate model weaknesses.
arXiv:2609.06229v1 Announce Type: cross Abstract: Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical...
The paper introduces Numbat, a self‑contained machine‑learning stack implemented entirely in Zig with no external runtime dependencies. It covers tensor computation, automatic differentiation, neural‑network modules, mixed precision, multi‑GPU training, data loading, and monitoring, and exposes a stable C ABI with over 1,400 entry points and bindings for six languages. The authors verify the stack against a reference implementation at multiple levels, uncovering ten silent recipe divergences, and demonstrate its practical capability by training a 25.9M‑parameter YOLOv8m detector on COCO 2017, achieving a competitive mAP score and matching single‑GPU performance.