arXiv:2608. 13459v1 Announce Type: cross Abstract: We address the use of large language models (LLMs) to help discover Isabelle proofs.
arXiv:2607. 00269v1 Announce Type: new Abstract: LLMs, solvers, and agent teams increasingly generate workflow actions, repairs, and plans, but a generated action may be syntactically valid yet stale, infeasible, conflicting, or destructive of the evidence that triggered a repair.
The study investigates how language models equipped with tools can still produce unsupported final claims, even when a single tool call could resolve the uncertainty. It defines two metrics—occurrence (how often unsupported claims arise) and conditional repair (how often they are fixed when evidence is provided). Experiments on Qwen3-32B and Gemma 4 show that providing the missing evidence consistently repairs all unsupported claims in the Qwen3-32B setup, while the Gemma 4 model never produced unsupported claims under the tested conditions.
arXiv:2608.23001v1 Announce Type: new Abstract: Automated manuscript pipelines often regenerate an entire section to repair a local defect, allowing unrelated metrics and citations to change even whe...
The paper investigates whether large language models (LLMs) can automatically verify no-code bug fixes—solutions that instruct users to change settings, upgrade versions, or alter workflows—by executing them in a real browser environment. Using 322 fixes from a benchmark, the study tests three executor agents (OpenCUA‑72B, Claude Sonnet 5, and Meta’s Muse Glimmer) and finds that only 17.6% of issues could be set up and passed sanity checks, with resolution rates ranging from 14.6% to 49.7% depending on the executor. Even the best configuration resolved fewer than half of the fixes, and executor choice significantly affected outcomes, highlighting the need for execution-based verification before such fixes reach users.
The paper introduces THEMIS, a stage-aware repair workflow that externalizes the requirement-to-repair process by generating semantic interpretations, a runtime requirement-code graph, graph-derived developer guidance, retained repair rationale and patches, and post-edit audit records. A retrospective audit of 300 SWE-bench Lite cases shows that these artifacts enable cross-stage inspection, with a complete developer rationale available for 288 cases and 214 cases retaining a full audited field set. The retained records also allow systematic measurement of cross-stage correspondence, revealing high recurrence of target symbols across rationales and patches, and a preliminary improvement in resolving cases compared to a direct same-input condition.
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
The paper introduces a reference‑free proxy‑judge framework for autoformalization, separating correctness into grounding, statement faithfulness, coverage, and inference. It presents three theoretical results linking proxy error rates to pass‑but‑wrong rates, the impact of false rejections on refinement, and a decomposition of final pass rates. The framework is instantiated with eight proxies and evaluated on ProofNet, miniF2F, e‑SNLI, and ProntoQA, showing that the predictions hold and identifying failure routes.
KAPPA is an open‑source toolkit that uses language‑model agents to translate LaTeX proofs from statistical papers into Lean 4, where each numbered result is verified step by step against the original text. The system assigns one of four verdicts—verified unconditionally, verified conditionally, issue recorded, or unfinished—to every result, and it can run unattended, re‑check only changed sections, and report resource usage. In tests on four example papers and three published statistical papers, KAPPA detected the known errors and additional issues that were later confirmed by hand, while a direct language‑model approach without the kit produced fewer correct verdicts and sometimes over‑claimed proofs.
arXiv:2610.39468v1 Announce Type: cross Abstract: AI coding agents now do long, multi-step and multi-session software work with little supervision. When a developer comes back to that work, the usefu...
Papers in statistical theory rest on long proofs that few readers check line by line. We present KAPPA (Kit of Agents for Proof-checking Papers with a proof Assistant), open-source software that organ...
The paper introduces RefactorProof, a benchmark that tests whether proofs generated for code remain valid after program refactoring that preserves behavior. Using 458 Lean-certified variants derived from 185 VERINA tasks, the study finds that structural refactorings expose brittleness in proofs, with task-normalized structural PSR ranging from 4.3% for supplied references to 13.3–26.8% for specialized provers. Despite higher failure rates, specialized prover proofs (DeepSeek-Prover-V2-7B, Goedel-Prover-V2-32B, Goedel-Prover-V2-8B) show significantly higher unchanged-proof survival compared to reference artifacts, and many failures can be repaired with minimal edits.
arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author mean...