arXiv AI
Aug 7
An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics
arXiv:2604. 15145v2 Announce Type: replace Abstract: The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task.
Beyond Correctness: Toward Automated Novelty Verification with Lean 4
arXiv:2608. 14669v1 Announce Type: new Abstract: Artificial intelligence systems applied to mathematics verify correctness but not novelty: an automatically generated theorem can compile in Lean without errors and yet be an already known result.
Related stories
arXiv AI
Jun 30
Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
arXiv AI
Sep 25
Learning to Discover Interesting Mathematics
The paper introduces a method for evaluating the intrinsic interestingness of mathematical theorems by comparing the length of their proofs to the length of their statements. It trains a 27B language model to predict proof difficulty, enabling the generation and selection of more interesting theorems while significantly reducing overlap with existing Mathlib. The approach allows iterative expansion of a self‑building, machine‑verified mathematical library guided by quantifiable metrics.
arXiv AI
Aug 18
Does the Proof Prove It That Way? Faithful Formalization of Elements Proofs
arXiv:2608. 15432v1 Announce Type: new Abstract: In formal verification, both the autoformalization of statements and automated proof search have been studied extensively.
arXiv AI
Jul 1
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
arXiv:2604. 19047v2 Announce Type: replace-cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity.
arXiv AI
Jun 16
SorryDB: Can AI Provers Complete Real-World Lean Theorems?
arXiv:2603. 02668v2 Announce Type: replace Abstract: We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub.
arXiv AI
Jun 11
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
arXiv:2606. 12071v1 Announce Type: cross Abstract: LLMs are increasingly used to generate and judge scientific ideas.
arXiv Computation and Language
Sep 1
MemoNoveltyAgent: A Historical Research Memory-Aware Agent Workflow for Paper Novelty Assessment
arXiv:2603.20884v4 Announce Type: replace Abstract: To alleviate the heavy burden of paper screening, researchers increasingly rely on existing AI agents, such as AI reviewers or DeepResearch, for pa...
arXiv AI
Aug 25
CausalSmith: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inference
CausalSmith is a framework that automates theoretical research in causal inference by integrating a Lean proof assistant with a self‑improving agentic pipeline. It uses Causalean, a Lean library of over 7,000 machine‑checked declarations, and a pipeline that selects topics, proposes results, formalizes statements, constructs proofs, and audits them against informal claims. The system’s artifacts and source code are publicly available on GitHub.
arXiv AI
Sep 12
NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment
NovGauge is a new benchmark designed to diagnose large language models’ ability to assess scientific paper novelty. It contains 619 paper pairs and 50 multi-paper sets, each labeled along three dimensions—task, problem, and method—by experts from ICLR reviewer overlap claims and survey co-citations. The study evaluates 18 LLMs, revealing high hallucination rates and weak evidence grounding, with the best model achieving only 43‑72% verified F1 across dimensions.
arXiv AI
Jun 9
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
arXiv:2606. 09450v1 Announce Type: new Abstract: LLMs have recently achieved strong results on formal proving benchmarks.
arXiv AI
Jul 2
Theoria: Rewrite-Acceptability Verification over Informal Reasoning States
arXiv:2607. 01223v1 Announce Type: new Abstract: When should an AI system's answer be trusted?