arXiv AI
The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Oct 26
SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks
arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...
arXiv AI
Oct 31
SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
SciGame is a new benchmark that transforms scientific literature into structured cards, each providing both an Original and a Counterfactual environment for the same task. It evaluates language‑model agents in identification and discovery scenarios under Prior‑only, Fixed, and Adaptive protocols, using historical canonical laws and a live temporal collection. Experiments with five advanced models reveal that prior knowledge can hurt performance when relations change, adaptive experimentation benefits strong models but not weak ones, and accurate explanations may mask quantitative errors, highlighting the difficulty of recovering exact symbolic laws.
arXiv Computation and Language
3d ago
A State Change Is Local, Its Consequences Are Not: Dependency-Aware State Revision for Long-Term Agent Memory
The paper introduces StateGraph, a system for dependency-aware state revision in long-term agent memory. It constructs an evolving state dependency graph from interaction histories and uses counterfactual verification to determine which dependent states should be invalidated when an upstream fact changes. Evaluation on the new StateChangeBench benchmark shows StateGraph outperforms a CUPMem baseline, suggesting that explicit dependency tracking can improve selective memory revision under changing conditions.
arXiv AI
Oct 9
What Output-Only Review Cannot Verify: Study Contracts for Research Agents
The paper introduces study contracts that bind declared experimental choices, run obligations, and claim scope to recorded execution evidence, aiming to distinguish contract-relative verification from scientific truth. A diagnostic using eight self‑authored clean/mutated pairs shows that a deterministic checker can detect all registered mutations when the fault‑specific rule is applied to approved and executed objects. In a broader test involving eighteen judge aliases and 144 mutated evaluation cases, 104 cases received defect flags, while the rest were abstentions or failures, illustrating the challenges of an information‑asymmetric development setting.
arXiv AI
Jun 9
SciTrace: Trajectory-Aware Safety Reasoning for Scientific Discovery Agents
arXiv:2606. 08234v1 Announce Type: new Abstract: LLM-based scientific agents have shown strong capacity for autonomous research, yet their safety layers remain structurally divorced from core reasoning: they inspect pipeline outputs rather than shaping the deliberation that produces them.
arXiv AI
Jul 14
FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.