arXiv AI
Jul 7
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
arXiv:2607. 04293v1 Announce Type: cross Abstract: Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention.
SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
SciGame is a new benchmark that transforms scientific literature into structured cards, each providing both an Original and a Counterfactual environment for the same task. It evaluates language‑model agents in identification and discovery scenarios under Prior‑only, Fixed, and Adaptive protocols, using historical canonical laws and a live temporal collection. Experiments with five advanced models reveal that prior knowledge can hurt performance when relations change, adaptive experimentation benefits strong models but not weak ones, and accurate explanations may mask quantitative errors, highlighting the difficulty of recovering exact symbolic laws.
Related stories
arXiv AI
Sep 2
EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation
arXiv:2609.01526v1 Announce Type: new Abstract: Scientific agents must learn not only how to reason, but also what to believe. However, existing LLM agents typically express scientific hypotheses in...
arXiv AI
Jun 4
FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
arXiv:2606. 04751v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents in scientific tasks.
arXiv AI
Aug 3
Scaling Scientific Discovery Environments for Turn-Level Agentic RL
arXiv:2607. 28990v1 Announce Type: new Abstract: Large language model agents have shown promising capabilities in data-driven scientific discovery tasks, where an agent interacts with an execution environment and produces a statistical claim.
arXiv AI
Jul 14
FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.
arXiv Machine Learning
Sep 2
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
arXiv:2605.26087v2 Announce Type: replace-cross Abstract: Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of...
arXiv AI
4d ago
PAPERBENCHX: Can AI Reproduce Science Across Disciplines?
PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.
arXiv AI
Jun 11
Can AI Agents Synthesize Scientific Conclusions?
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
arXiv Computation and Language
Oct 9
SAIL: Scientific Agentic Intelligence via a Science-Aware Loop
SAIL is an open model with 35 B total and 3 B active parameters designed for literature research, scientific coding, and multi‑step research workflows. It is developed through a science‑aware improvement loop in which agents built on frontier AI models analyze task failures, create training tasks to address capability gaps, and iterate this process across multiple development cycles. The training pipeline combines supervised fine‑tuning, specialist training, multi‑teacher on‑policy distillation, and agentic reinforcement learning, enabling SAIL to achieve competitive performance on scientific research tasks with fewer parameters than leading open‑weight models.
Hugging Face Trending Papers
Sep 24
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
ExplorationBench is a benchmark designed to evaluate AI systems’ ability to conduct scientific exploration in verifiable alien worlds, where rules are executable and can be precisely checked. It consists of two sandboxes—AlienCode and AlienLogic—each offering discovery targets, tasks, flawed manuals, environmental feedback, and tool‑call schemas. The benchmark tests whether systems can generate new hypotheses, design experiments, and iterate on results, rather than merely recalling pre‑trained knowledge, and finds that top performers can acquire and apply unfamiliar rules, though performance varies across exploration trajectories.
arXiv AI
Sep 25
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
ExplorationBench is a new benchmark designed to evaluate AI systems’ ability to conduct scientific exploration in verifiable alien worlds. It comprises two sandbox environments—AlienCode and AlienLogic—each containing discovery targets, tasks, flawed manuals, and tool‑call schemas that force systems to formulate hypotheses, design experiments, and iterate on results. Ten AI systems were tested, revealing that while the best performers can learn and apply unfamiliar rules, their progress varies across exploration trajectories and can even regress with continued exploration.
arXiv AI
Oct 27
DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
arXiv:2609.33357v2 Announce Type: replace Abstract: Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially red...