TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
By Zhibo Yang, Chen Zhang, Yuewei Zhang, Hao Wang
arXiv:2610.40906v1 Announce Type: new
Abstract: Recent scientific agents build knowledge recursively by retrieving evidence, constructing an intermediate state, and judging whether that state contain...
By Lun Ai, Yu-Heng Wu, Jonathan Schaul, Leonie K\"uchenhoff, Denes Turei, Francesco Carli, Fabio Petroni, Aurelien Dugourd, Julio Saez-Rodriguez
arXiv:2606. 09556v1 Announce Type: new Abstract: AI Scientist agents are often evaluated as if capability were mainly a function of model quality, prompting, or reasoning scaffolds.
By Yinan Wang
The paper reports that a model can pass fidelity checks—verifying that extracted values match the source—without actually opening a datasheet, due to a hidden constraint that disables tool use. To address this, the authors log every tool call in an agentic benchmark and develop two instruments: a rule‑based failure‑attribution classifier and a silent‑failure detector that flags runs based solely on which tools were invoked. While the detector shows low false positives on clean extractions and recovers all planted faults, its recall against correct tool usage but incorrect answers remains unmeasured, and a partial causal chamber confirms only a subset of claims, highlighting limitations in physical verification.
By Qing Ye, Meng-Hsuan Lin
The paper introduces LLM-IDEA, an identifiability‑driven experimental agent that guides large language model scientists in closed‑loop discovery of mechanistic world models. It incorporates an identifiability engine that classifies experimental plateaus as capability limits, resolvable within the design class, or certified exhausted, and demonstrates its effectiveness on benchmarks such as ODEBench, DiscoverPhysics, and a custom Alien Universe testbed. The agent’s ability to distinguish when further experimentation is futile versus when new designs are needed improves the efficiency and reliability of autonomous scientific discovery.
By Surya Shetty, Ulisses Braga-Neto
LLM-IDEA is an identifiability‑driven experimental agent that guides large language model scientists in autonomous discovery of mechanistic world models. It evaluates whether a plateau in learning is due to the agent’s capability limits, the model’s resolvability within the design class, or a certified exhaustion of the experiment space. The system demonstrates its effectiveness across benchmarks such as ODEBench, DiscoverPhysics, and a custom Alien Universe testbed, consistently identifying when additional experiments are futile and when new experimental designs are required.