TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
By Zhibo Yang, Chen Zhang, Yuewei Zhang, Hao Wang
arXiv:2610.40906v1 Announce Type: new
Abstract: Recent scientific agents build knowledge recursively by retrieving evidence, constructing an intermediate state, and judging whether that state contain...
By Lun Ai, Yu-Heng Wu, Jonathan Schaul, Leonie K\"uchenhoff, Denes Turei, Francesco Carli, Fabio Petroni, Aurelien Dugourd, Julio Saez-Rodriguez
arXiv:2606. 09556v1 Announce Type: new Abstract: AI Scientist agents are often evaluated as if capability were mainly a function of model quality, prompting, or reasoning scaffolds.
By Yinan Wang
The paper reports that a model can pass fidelity checks—verifying that extracted values match the source—without actually opening a datasheet, due to a hidden constraint that disables tool use. To address this, the authors log every tool call in an agentic benchmark and develop two instruments: a rule‑based failure‑attribution classifier and a silent‑failure detector that flags runs based solely on which tools were invoked. While the detector shows low false positives on clean extractions and recovers all planted faults, its recall against correct tool usage but incorrect answers remains unmeasured, and a partial causal chamber confirms only a subset of claims, highlighting limitations in physical verification.
By Qing Ye, Meng-Hsuan Lin
The paper introduces LLM-IDEA, an identifiability‑driven experimental agent that guides large language model scientists in closed‑loop discovery of mechanistic world models. It incorporates an identifiability engine that classifies experimental plateaus as capability limits, resolvable within the design class, or certified exhausted, and demonstrates its effectiveness on benchmarks such as ODEBench, DiscoverPhysics, and a custom Alien Universe testbed. The agent’s ability to distinguish when further experimentation is futile versus when new designs are needed improves the efficiency and reliability of autonomous scientific discovery.
By Surya Shetty, Ulisses Braga-Neto
LLM-IDEA is an identifiability‑driven experimental agent that guides large language model scientists in autonomous discovery of mechanistic world models. It evaluates whether a plateau in learning is due to the agent’s capability limits, the model’s resolvability within the design class, or a certified exhaustion of the experiment space. The system demonstrates its effectiveness across benchmarks such as ODEBench, DiscoverPhysics, and a custom Alien Universe testbed, consistently identifying when additional experiments are futile and when new experimental designs are required.
The paper introduces SCILAWS-BENCH, a benchmark for evaluating large language models (LLMs) on scientific law discovery. It contains 118 problems from 381 scientific papers, covering 291 candidate laws and about 8 million real data points across six disciplines. The benchmark offers two settings: SCILAWS-REAL, where models must propose laws from fixed real observations, and SCILAWS-PARALLEL, where models actively query synthetic worlds to recover hidden laws.
By Yiming Huang, Ziche Liu, Zhuohang Wu, Yiqian Wang, Junxia Cui, Xinkai Zou, Linjun Mao, Nan Huang, Naicheng Yu, Kaijie Zhu, Yue Ma, Kun Zhou, Letian Peng, Jingbo Shang
ULTRADISCOVERY is a benchmark that tests abductive exploration in an epistemically open, interconnected universe. It features five domains where agents must revise an initial theory and predict outcomes of unseen cross‑domain interventions, with a $2 imes 2$ design manipulating representation openness and evidence alignment. Experiments show that agents struggle to compose evidence into transferable representations, especially when the representation is open, and that even with aids exact predictions are rare without large budgets.
By Weihan Li, Tianshi Zheng, Yangqiu Song, Ginny Y. Wong, Simon See
The paper proposes a framework for studying how alignment difficulty scales with model size, treating alignment as a family of measurable scaling relations for different risk categories. It introduces operational definitions of burden, distinguishes observed, audited, and true alignment, and proves that the largest exponent among corrected risks determines long‑run behavior. The authors preregistered and applied the protocol to public data, finding scaling exponents for attack success, truthfulness, dispositions, sycophancy, and a backdoor, and released interactive browser games to illustrate the laws.
By Jeremy Canale
arXiv:2608.28997v1 Announce Type: new
Abstract: In May 2026 an OpenAI model produced a counterexample to the Erd\H{o}s unit distance conjecture. Five mathematicians published a human-verified version...
By Maher Kallel, Mohamed El Louadi
arXiv:2610.20457v1 Announce Type: new
Abstract: Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, wr...
By Delip Rao, Ahmed Muharram, Chris Callison-Burch
arXiv:2610.14843v1 Announce Type: new
Abstract: Claims of recursive self-improvement are easy to make and hard to falsify: did improvement come from re-using designer-provided mechanisms, or from mec...
By Wenbin Mao, Pei Zou