arXiv:2608.30581v1 Announce Type: new
Abstract: Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an...
By Dennis Gross, Helge Spieker
arXiv:2608.17795v2 Announce Type: replace
Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at in...
By Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee, Vasu Rangarajan, Viji Krishnamurthy
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
The paper introduces "code explanations"—programs that describe how large language models (LLMs) generate outputs across a dataset, enabling automatic verification through execution. It defines a Consistency metric to assess how often a program reproduces the LLM’s outputs and evaluates this approach on three open‑source code‑capable LLMs across eight tasks, finding that code explanations are significantly shorter than chain‑of‑thought explanations and often better align with model behavior. However, consistency drops on tasks with complex linguistic components, and even with consistency feedback and execution, LLMs struggle to produce reliable explanations for these harder tasks.
By Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath
The paper introduces VERA, an automated framework that audits large language model (LLM) reasoning in software vulnerability analysis. Instead of relying on free‑form explanations, VERA requires models to produce a Structured Reasoning Record (SRR) that captures pointers, memory operations, and state transitions in machine‑readable fields. A multi‑stage judge then checks each SRR against eight reasoning failure modes, revealing that reasoning flaws are as common in correct verdicts as in incorrect ones and that VERA detects 87% of errors missed by free‑form LLM‑as‑judge evaluations.
arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.
By Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou, Weibin Lin, Zheng Zheng
HoarePrompt is a new method that applies program verification concepts to natural language requirements, using large language models to generate step‑by‑step natural language descriptions of program states. It incorporates a few‑shot k‑induction technique to handle loops and then evaluates whether the annotated program satisfies the requirements. On the CoCoClaNeL dataset, HoarePrompt raises the Matthews correlation coefficient by 61% over zero‑shot chain‑of‑thought prompts and by 106% over test‑generation classifiers, with the inductive reasoning component adding a 26% MCC improvement.
By Dimitrios Stamatios Bouras, Yihan Dai, Tairan Wang, Yingfei Xiong, Sergey Mechtaev
arXiv:2607. 11342v1 Announce Type: cross Abstract: Despite their central role in fault detection, test oracles remain challenging to construct effectively.
By Yue Zhao, Binish Tanveer, Jelena Zdravkovic
arXiv:2606. 19588v1 Announce Type: new Abstract: Formal tools such as SAT and SMT solvers are increasingly embedded in language model reasoning pipelines when a safety or security critical question can be formulated in logic.
By Zunchen Huang, Songgaojun Deng
Formal tools such as SAT and SMT solvers are increasingly embedded in language model reasoning pipelines when a safety or security critical question can be formulated in logic. Unlike chain of thought whose steps are sampled from the model distribution without formal guarantee, a solver produces a sound and independently verifiable answer.
arXiv:2606. 03601v1 Announce Type: cross Abstract: While safety alignment and guardrails help large language models (LLMs) avoid harmful outputs, they can also induce overrefusal, i.
By Qinyan Zhou, Peixin Zhang, Jun Sun, Haonan Zhang, Dongxia Wang
arXiv:2607.14109v2 Announce Type: replace
Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central...
By Inder Preet, Shuxin Lin, Dhaval Patel