Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
The Flow has not summarised this story yet — read it at arXiv AI.
Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an action was chosen. However, LLMs often generate p...
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.
arXiv:2608.17795v2 Announce Type: replace Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at in...
The paper introduces "code explanations"—programs that describe how large language models (LLMs) generate outputs across a dataset, enabling automatic verification through execution. It defines a Consistency metric to assess how often a program reproduces the LLM’s outputs and evaluates this approach on three open‑source code‑capable LLMs across eight tasks, finding that code explanations are significantly shorter than chain‑of‑thought explanations and often better align with model behavior. However, consistency drops on tasks with complex linguistic components, and even with consistency feedback and execution, LLMs struggle to produce reliable explanations for these harder tasks.
arXiv:2607. 11342v1 Announce Type: cross Abstract: Despite their central role in fault detection, test oracles remain challenging to construct effectively.