← Back to all news
arXiv AI By Zengji Tu, Xiang Liu, Jiarui Sun, Ruoyu Wu, Zhihui Qi, Chengda Wu, Yutao Zhou, Bo Yang, Ziyan Zhang, Jialong Wu, Liang Chen, Kuan Li, Steven Dillmann, Yuanqi Du, Soojung Yang, Shizhe Diao, Jianjun Wu, Junlang Liu, Xuming Hu, Lin Chang, Baobao Chang, Xiaowen Chu, Philip S. Yu, Tong-Yi Zhang, Hao Cheng, Qin Han, Yanye Feng, Chenru Duan, Chuncheng Liu, Yan Liu, Yuchen Gao, Zhao Li, Ronghua Zhang, Tory Xu, Renjian Song, Yaokai Fu

Science Prediction Universe

Read the original on arXiv AI →

The paper introduces the Science Prediction Universe (SPU), a benchmark comprising 100 public verified tasks and 16 private wet‑lab tasks across chemistry, materials science, biology, and medicine, designed to test agents’ ability to predict experimental outcomes. Thirteen agent configurations using twelve frontier models were evaluated, revealing significant performance gaps: Spearman correlations ranged from 0.227 to 0.478 on the public set. Analysis of over a thousand trajectories indicates that current agents struggle more with scientific judgment than tool execution, often relying on shared priors and trivial proxies rather than accurate predictions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 31

PAPERBENCHX: Can AI Reproduce Science Across Disciplines?

PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.

By Ruoyu Wu, Zengji Tu, Aolong Sun, Tianyi Ma, Junren Li, Xinbo Xu, Li Fu, Ziyan Zhang, Pinrui Huang, Xiaochen Wang, Chengda Wu, Yutao Zhou, Jianjun Wu, Lin Chang, Baobao Chang, Liang Chen, Kuan Li
agentsroboticsbenchmarks
More like this →
Hugging Face Trending Papers
Oct 13

Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery

The paper introduces Autonomous Science Bench, a benchmark for AI agents that must choose experiments under a limited budget. It features six real-world design-build-test-learn campaigns in biology, chemistry, and materials science, where agents select design batches, receive lab measurements, and iteratively refine a predictive model. Success is measured by two calibrated gates—discovery and learning—highlighting distinct agent weaknesses that overall success rates alone would mask.

agentsbenchmarks
More like this →
arXiv AI
Oct 2

EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights

EurekaBench is a cross‑domain benchmark designed to assess AI agents’ capacity to discover new scientific insights. It comprises 26 long‑horizon tasks spanning fields such as neuroscience, chemistry, and astrophysics, with 306 expected insights that mechanisms should support. The evaluation measures adherence to scientific constraints, predictive accuracy, and the ability to generate insights or guide future research, revealing that current AI agents excel at prediction but lag in producing meaningful scientific insights.

By Jiayi Geng, Zhengxuan Wu, Kevin S. Chen, Seungone Kim, Joseph Janssen, Zora Zhiruo Wang, Bhupalee Kalita, Runtian Gao, Aaron Ho, Andrew Oakleigh Nelson, Olexandr Isayev, Francisco Villaescusa-Navarro, Ching-Yao Lai, Howard Chen, Graham Neubig
agentsbenchmarks
More like this →