The paper examines how AI‑written research papers often contain misattributed numerical claims. By re‑analyzing 540 judgments from a benchmark of nine AI agents on fifteen research problems, the authors manually labeled 256 claims and found that most failures stem from misalignment between the paper and the underlying work rather than outright cheating. They propose a deterministic claim‑to‑evidence checker that achieves over 90% precision and, when combined with an LLM verifier, offers a more reliable audit of AI‑generated research.
arXiv:2610.20457v1 Announce Type: new Abstract: Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, wr...
arXiv:2607. 26512v1 Announce Type: new Abstract: AI agents can draft claims faster than authors can check whether the cited or retrieved evidence supports them.
arXiv:2609.15319v1 Announce Type: cross Abstract: Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced a...
arXiv:2609.10293v1 Announce Type: new Abstract: In high-stakes domains such as legal practice, a language-model answer is only useful to the extent that a reader can verify each claim against the sou...
VeriHarness is a method that enhances verification for large language model agents tackling long‑horizon tasks without needing reference answers at test time. It transforms the base LLM into an agentic verifier by providing a workspace, evidence tools, and reusable verification skills, using disagreement resolution and consensus challenge to evaluate competing claims. Across five benchmarks and two frontier models, VeriHarness outperforms baselines, achieving significant performance gains and demonstrating self‑improvement of verification skills from failure feedback.
arXiv:2607. 01223v1 Announce Type: new Abstract: When should an AI system's answer be trusted?
ClaimScope is a 72‑task benchmark that audits mathematical claims by requiring a verdict, scope, counterexample, and repair, covering areas such as optimizer‑state transport and exact $2 imes2$ linear algebra. The study evaluates Qwen3‑8B using a witness checker that only examines the model’s submitted state, showing that verifier feedback improves counterexample acceptance and safe completions compared to generic feedback. The results demonstrate that separating claim classification from executable evidence can yield gains in witness‑structure correction without necessarily improving broader reasoning.
TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
arXiv:2607. 18240v1 Announce Type: new Abstract: Large language models (LLMs) can achieve strong fact-checking accuracy, yet forced binary decisions conceal a critical reliability problem: systems may issue confident verdicts even when supporting evidence is weak, sparse, or internally inconsistent.
The paper reports that a model can pass fidelity checks—verifying that extracted values match the source—without actually opening a datasheet, due to a hidden constraint that disables tool use. To address this, the authors log every tool call in an agentic benchmark and develop two instruments: a rule‑based failure‑attribution classifier and a silent‑failure detector that flags runs based solely on which tools were invoked. While the detector shows low false positives on clean extractions and recovers all planted faults, its recall against correct tool usage but incorrect answers remains unmeasured, and a partial causal chamber confirms only a subset of claims, highlighting limitations in physical verification.
arXiv:2607. 12650v1 Announce Type: cross Abstract: Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions need not hold up under formal scrutiny.
arXiv:2610.17724v1 Announce Type: new Abstract: Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across t...