The paper introduces four executable operations—identifying redundancies (L1), coarse‑graining dynamics (L2), transferring structure (L3), and inferring missing quantities from conservation imbalances (L4)—to modify physical descriptions in automated scientific discovery systems. It evaluates these operations alongside three interpretability settings (null acceptance, positive acceptance, consequence agreement) on chaotic systems, controlled benchmarks, language and protein models, and sparse autoencoders, revealing that descriptive tests can accept empty candidates and accepted candidates may lack verified consequences. The study demonstrates that reliable automated discovery must assess both what a test accepts and the scientific consequences it establishes.
arXiv:2607.27617v2 Announce Type: replace Abstract: Identical language-model answers can arise from hidden states that support different future computations, so current-answer probes do not establish...
arXiv:2608. 12935v1 Announce Type: new Abstract: Perturbation methods explain model decisions by measuring prediction changes under altered inputs, but response magnitude tells us only how much a model reacts, not what that reaction means.
The paper introduces an experimental model‑class revision framework that jointly proposes structural edits to a hypothesis space and diagnostic experiments to test those edits. By coupling a class‑level distinguishability objective with anytime‑valid sequential evidence, the method only revises the model class after the current one is rejected. On 400 controlled dynamical environments, the approach achieves 89.5% exact recovery with 32 experiments, outperforming baselines and transferring well to unseen mechanisms, library insufficiency detection, and other benchmark tasks.
arXiv:2609.00654v1 Announce Type: new Abstract: We describe the SciTrue team's participation in both subtasks of the NTCIR-19 SciClaimEval task~\cite{sciclaimeval}, which asks systems to verify scien...
The paper introduces CELLAUDIT, a method for auditing whether inputs claimed to influence predictive models actually do so. By testing if an input can enter the computation, whether predictions depend on it, and if that dependence improves observed responses, the authors evaluate agent-generated predictors on a morphology‑transcriptomics benchmark (BBBC047). Their findings show that many models claim compound contributions that are not supported by the data, and that falsification‑guided revisions can recover genuine input effects while improving performance.
arXiv:2608. 05670v1 Announce Type: new Abstract: A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness.
arXiv:2610. 02860v1 Announce Type: new Abstract: Low latent prediction error does not establish that a world model distinguishes the consequences of its actions.
arXiv:2607. 26929v1 Announce Type: cross Abstract: The same diagnostic result can support or challenge one causal claim yet fail to address another when the claims concern different populations, outcomes, estimands, pathways, or identifying assumptions.
The paper investigates mechanistic interpretability, focusing on how automated circuit discovery is evaluated. It shows that the commonly used faithfulness objective can favor circuits that reproduce a model’s behavior poorly, creating an objective-level recovery gap. Experiments on four human-reference tasks and InterpBench reveal that many discovery methods misrank candidate circuits, and that restoring excluded signals can correct most of these misrankings without altering the circuits’ behavior.
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
arXiv:2605. 10246v2 Announce Type: replace Abstract: AI scientist systems are increasingly deployed for autonomous research, yet their academic integrity has never been systematically evaluated.
arXiv:2608. 05702v1 Announce Type: new Abstract: Scientific machine learning commonly validates models at the level of a subdomain, a benchmark split, or an explanation for one prediction.