The paper investigates Transformer Programs (TP), a constrained Transformer architecture that outputs readable Python code. It shows that token‑level accuracy can mask incorrect full‑sequence outputs, and that inspecting the generated programs reveals failures missed by standard tests. By using program inspection to guide data augmentation, the authors achieved higher sequence accuracy on several algorithmic tasks, demonstrating the value of program‑level analysis for training TP models.
arXiv:2608.29376v1 Announce Type: new Abstract: Code generation increasingly relies on large transformer models, whose capability advances with scale. Yet such a scale is costly, creating demand for...
arXiv:2509. 23806v2 Announce Type: replace-cross Abstract: Concolic testing for neural networks alternates concrete execution with constraint solving to search for inputs that flip model decisions.
arXiv:2512. 01870v3 Announce Type: replace Abstract: We study whether a transformer network can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers.
arXiv:2604.13082v3 Announce Type: replace-cross Abstract: Large language models today still make mistakes on multi-digit arithmetic when they do not have tools, and part of the reason is how numbers...
arXiv:2510. 04212v4 Announce Type: replace-cross Abstract: The pursuit of computational efficiency has driven the adoption of low-precision formats for training transformer models.
arXiv:2605.03953v3 Announce Type: replace-cross Abstract: Recent Transformer architectures expose later layers to representations computed in earlier layers, motivated by the observation that low-lev...
arXiv:2610.08258v1 Announce Type: new Abstract: Auditing the claimed outcomes of large language model (LLM) training is challenging when model weights and training data are private, while cryptograph...
arXiv:2602. 08857v2 Announce Type: replace-cross Abstract: Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages.
arXiv:2510. 14331v3 Announce Type: replace Abstract: We study program-learning methods that are efficient in both samples and computation.
arXiv:2610.09003v1 Announce Type: new Abstract: Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and...
The paper proposes InfiLoop, a loop-native residual connection designed for looped Transformers to prevent performance degradation as iterations increase. InfiLoop learns which past computations to retain and how much to accept from each new update, using content-based weighting and learned temporal decay to maintain a running summary of recurrent states. The approach yields strong results on reasoning tasks, achieving 97.9% exact accuracy on Sudoku-Extreme and 13.6% pass@2 on ARC-AGI-2, and continues to improve with over 20,000 test-time iterations.
arXiv:2606. 26492v1 Announce Type: cross Abstract: Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task.