The paper introduces a roundtrip verification method for ensuring that large language models (LLMs) produce faithful formalizations of natural language statements. By formalizing a statement, translating it back to natural language, re-formalizing, and checking logical equivalence with a formal tool, the approach detects inconsistencies without needing ground-truth annotations. When inconsistencies are found, a diagnosis localizes the error to a specific translation step, and a scoped repair operator attempts to correct it. The framework is evaluated on the Texas Transportation Code and Texas Parks and Wildlife Code using Claude Opus and GPT-5, showing that diagnosis-guided scoped repair is most effective and that rules failing the equivalence check exhibit significantly more natural language inference drift.
arXiv:2607. 17641v1 Announce Type: new Abstract: Verify-repair loops are a standard means for large language model (LLM) agents to correct faulty plans in code generation, mathematical reasoning, and tool use.
The paper presents an end‑to‑end pipeline for translating natural language planning descriptions into PDDL problem instances using large language models. It incorporates multiple checks—syntactic parsing, planner success, domain conformance, an LLM critic, and iterative repair—to ensure faithfulness to the original task. Experiments on Planetarium, AutoPlanBench, and curated PDDL~2.1 problems reveal that operational success can diverge from benchmark‑reference reconstruction, and that structured repair improves outcomes while PDDL~2.1 remains challenging for reference reconstruction.
Verify-repair loops are a standard means for large language model (LLM) agents to correct faulty plans in code generation, mathematical reasoning, and tool use. When both the verifier and the repairer are noisy, repair can damage already-correct plans, and reported acceptance keeps rising while true validity falls, so existing methods lack a principled basis for deciding when repair should stop.
The paper introduces A-CEGIS, a lightweight framework that employs counterexamples as feedback to evaluate and improve multi-turn natural-language-to-regex synthesis. In experiments on 30 NL-RX-Turk tasks, counterexample feedback enables agents to solve 90% of tasks within four turns, outperforming zero‑shot generation, generic self‑correction, and error‑only feedback. A full diagnostic run with hardening solves all hidden tasks by the final turn, achieving a mean time‑to‑success of 2.7 turns and robust success of 77% after targeted probing.
SkillForge is a framework that breaks down formal code synthesis into reusable atomic skills, each handling a specific subtask such as specification inference, body synthesis, invariant generation, error diagnosis, or repair. A verification-driven harness coordinates these skills by submitting candidates to the Dafny verifier, diagnosing failures, and routing them deterministically to the appropriate repair skill until correctness is achieved or a budget is reached. On a curated benchmark, SkillForge outperforms state‑of‑the‑art agentic and iterative baselines, requiring fewer tokens and lower latency, with ablation studies showing each skill’s measurable contribution and rapid convergence.
FaithSieve is a Lean‑assisted framework that fine‑grains natural‑language mathematical proofs into local reasoning units, extracts typed proof obligations, and verifies them with formal evidence gated by semantic alignment. It introduces two expert‑verified datasets—ProofLoc‑Olympiad and ProofLoc‑University—to benchmark first‑error localization. On these benchmarks, FaithSieve outperforms direct‑judging baselines, achieving 81.43% and 84.5% exact first‑error accuracy respectively.
arXiv:2606. 03331v1 Announce Type: cross Abstract: Consumer device repair is an important but underexplored testbed for large language models (LLMs).
Consumer device repair is an important but underexplored testbed for large language models (LLMs). Repair tasks require reasoning over incomplete problem descriptions, hardware-specific diagnostics, actionable troubleshooting, and safety-critical decisions, where incorrect advice can cause device damage, battery hazards, or permanent data loss.
SysML v2's textual syntax enables compiler-based validation of model structure and language conformance. However, semantic mistakes that preserve syntactic validity but violate domain rules cannot be detected through compilers.
arXiv:2511. 09008v2 Announce Type: replace-cross Abstract: Large Language Models perform well at natural language interpretation and reasoning, but their lack of formal correctness guarantees limits their adoption in regulated industries like finance and health-care that operate under strict policies.
arXiv:2607. 13303v1 Announce Type: cross Abstract: Formal contracts are essential for software testing and verification, yet writing them remains labor-intensive and error-prone.
arXiv:2606. 24124v1 Announce Type: new Abstract: Multi-step reasoning with Chain-of-Thought (CoT) prompting remains fragile: logical errors or hallucinations in early steps silently propagate, producing confident but incorrect conclusions.