arXiv:2609.37233v1 Announce Type: cross
Abstract: Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require...
By Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang, Peisen Yao
arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.
By Soham Dan
Quasar is a new programming language designed to improve large language model (LLM) code actions by separating internal program logic from external tool calls. It allows developers to annotate external calls with effect information and modify internal execution to track these effects, enabling easier implementation of new features. The authors demonstrate Quasar’s utility by adding access control, autoparallelization, and conformal prediction for uncertainty quantification.
By Stephen Mell, Botong Zhang, David Mell, Shuo Li, Ramya Ramalingam, Nathan Yu, Stephan Zdancewic, Osbert Bastani
arXiv:2606. 14935v1 Announce Type: new Abstract: Frontier reasoning-tuned language models still fail on deductive tasks at depth, and the cost of improved performance through extended internal reasoning scales poorly.
By Agnieszka Mensfelt, Adarsh Prabhakaran, Adrian Haret, Vince Trencsenyi, Kostas Stathis
HERO is a method that improves large language models for program generation by combining offline and test‑time harness optimization. It learns reusable prompt updates from execution‑verified successes and failures, validates candidate revisions before adoption, and then refines each generated code instance using execution feedback. On four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, raising pass@1 by 7.6 points over the initial prompt and adding another 2.0 points through test‑time refinement while improving execution efficiency.
By Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati