arXiv AI
Jun 2
FVSpec: Real-World Property-Based Tests as Lean Challenges
arXiv:2606. 01008v1 Announce Type: cross Abstract: We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks.
Why3-py: A Tool for Formal Verification of Hypothesis Testing and Meta-Analysis in Python
arXiv:2607. 03951v1 Announce Type: cross Abstract: The reproducibility crisis in scientific research has received widespread recognition, thereby increasing the importance of meta-analyses that integrate statistical analyses from multiple studies.
Related stories
arXiv AI
Jul 28
SetGo: Metadata Readiness for Scientific AI Datasets
arXiv:2607. 22677v1 Announce Type: cross Abstract: Scientific datasets intended for AI use require both computational readiness for model training and metadata readiness for discovery, sharing, and reuse.
arXiv AI
Jun 2
scicode-lint: Detecting Methodology Bugs in Scientific Python Code with LLM-Generated Patterns
arXiv:2603. 17893v2 Announce Type: replace-cross Abstract: Methodology bugs in scientific Python code produce plausible but incorrect results that traditional linters and static analysis tools cannot detect.
arXiv AI
Jul 13
OpenProver: Agentic and Interactive Theorem Proving with Lean 4
arXiv:2607. 09217v1 Announce Type: new Abstract: In this system paper, we present OpenProver, an open-source system for LLM-driven automated theorem proving (ATP) with integrated Lean 4 formal verification.
arXiv AI
Jun 30
meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis
arXiv:2606. 28363v1 Announce Type: cross Abstract: Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points.
arXiv Machine Learning
Jul 28
MioFFAn: an Annotation Software for Formula Formalization with LLM Automation Capabilities
arXiv:2607. 22552v1 Announce Type: cross Abstract: The automatic translation of mathematical expressions in scientific literature into executable symbolic code (a process we refer to as Formula Formalization) is hindered by a severe scarcity of high-quality, ground-truth datasets specialized for technical scientific domains.
arXiv AI
Jun 30
Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
arXiv AI
Jul 3
Aria: An Agent For Retrieval and Iterative Auto-Formalization via Dependency Graph
arXiv:2510. 04520v2 Announce Type: replace Abstract: Accurate auto-formalization of theorem statements is essential for advancing automated discovery and verification of research-level mathematics, yet remains a major bottleneck for LLMs due to hallucinations, semantic mismatches, and their inability to synthesize new definitions.
arXiv AI
Jun 2
PBT-Bench: Benchmarking AI Agents on Property-Based Testing
arXiv:2605. 15229v3 Announce Type: replace-cross Abstract: Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue.
arXiv Machine Learning
Jul 27
CausalForge: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inference
arXiv:2607. 22511v1 Announce Type: cross Abstract: Automating theoretical research is constrained not only by the generation of candidate results, but also by their reliable evaluation.
arXiv AI
Jul 13
Diversifying to Verify: When Task-Equivalent Programs Differ in Verifiability
arXiv:2607. 09366v1 Announce Type: cross Abstract: Program verification is crucial for software correctness, but producing fully verified programs remains difficult in practice.
arXiv AI
Jun 9
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.