Hugging Face Trending Papers

Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle

arXiv AI
Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
arXiv AI
3d ago

Code as Explanations: Executable and Verifiable LLM Explanations

The paper introduces "code explanations"—programs that describe how large language models (LLMs) generate outputs across a dataset, enabling automatic verification through execution. It defines a Consistency metric to assess how often a program reproduces the LLM’s outputs and evaluates this approach on three open‑source code‑capable LLMs across eight tasks, finding that code explanations are significantly shorter than chain‑of‑thought explanations and often better align with model behavior. However, consistency drops on tasks with complex linguistic components, and even with consistency feedback and execution, LLMs struggle to produce reliable explanations for these harder tasks.

By Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath
Hugging Face Trending Papers
Oct 5

Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning

The paper introduces VERA, an automated framework that audits large language model (LLM) reasoning in software vulnerability analysis. Instead of relying on free‑form explanations, VERA requires models to produce a Structured Reasoning Record (SRR) that captures pointers, memory operations, and state transitions in machine‑readable fields. A multi‑stage judge then checks each SRR against eight reasoning failure modes, revealing that reasoning flaws are as common in correct verdicts as in incorrect ones and that VERA detects 87% of errors missed by free‑form LLM‑as‑judge evaluations.

arXiv AI
Sep 10

HoarePrompt: Structural Reasoning About Program Correctness in Natural Language

HoarePrompt is a new method that applies program verification concepts to natural language requirements, using large language models to generate step‑by‑step natural language descriptions of program states. It incorporates a few‑shot k‑induction technique to handle loops and then evaluates whether the annotated program satisfies the requirements. On the CoCoClaNeL dataset, HoarePrompt raises the Matthews correlation coefficient by 61% over zero‑shot chain‑of‑thought prompts and by 106% over test‑generation classifiers, with the inductive reasoning component adding a 26% MCC improvement.

By Dimitrios Stamatios Bouras, Yihan Dai, Tairan Wang, Yingfei Xiong, Sergey Mechtaev
Hugging Face Trending Papers
Jun 17

Analyzing the Narration Gap in LLM-Solver Loops

Formal tools such as SAT and SMT solvers are increasingly embedded in language model reasoning pipelines when a safety or security critical question can be formulated in logic. Unlike chain of thought whose steps are sampled from the model distribution without formal guarantee, a solver produces a sound and independently verifiable answer.