IdeaTaste is a benchmark that evaluates large language models (LLMs) on their ability to generate research proposals that align with what human researchers actually publish. The study compares proposals generated by six contemporary LLMs against real papers from four research areas, finding that LLMs achieve only about 26% of the citation matching and 18% of the idea matching of human communities. Even larger LLM populations and richer field contexts do not close this gap, indicating that LLMs struggle more with translating research directions into concrete approaches than with selecting the right topics.
arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...
Ideation Arena is a battle-style platform that evaluates research ideas generated by large language models (LLMs) and research agents through pairwise human assessment. The system builds shared literature contexts, collects over 6,000 double-blind comparisons from 105 computer science researchers, and constructs an Elo rating leaderboard to rank proposal-stage expert preferences. It also introduces Ideation Arena Eval, a benchmark to test whether automated evaluators align with human preferences, finding that current LLM judges achieve at best 72.56% Soft Accuracy on overall quality.
arXiv:2510. 24891v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated significant potential to accelerate scientific discovery as valuable tools for analyzing data, generating hypotheses, and supporting innovative approaches in various scientific fields.
The paper argues that progress in AI is not simply a function of scale or the breadth of ideas, but depends on diverse attention, which can be depleted by collective focus on a single paradigm. Analyzing 36 years of computer‑science literature, the authors find that after the Transformer, the field’s attention narrowed sharply, with many researchers shifting into AI and reviewer confidence dropping. Each major breakthrough— from AlexNet to large language models—raises the field’s capacity for ideas but also accelerates the exhaustion of those ideas, a cycle mirrored in structural biology after cryo‑EM and AlphaFold. The study warns that policies like reciprocal reviewing may unintentionally concentrate attention on prolific authors, urging institutions to safeguard diverse attention rather than reward concentration on a single winning paradigm.
arXiv:2603.01092v3 Announce Type: replace-cross Abstract: Scientific discovery is constrained not only by what is true, but by what is cognitively available to researchers. Many directions are cohere...
SoundnessBench is a benchmark of 1,099 real research proposals designed to test whether large language models can judge the methodological viability of a research idea before resources are invested. The study evaluates 12 frontier LLMs and finds a pervasive optimism bias, with standard prompting yielding a 74.0% false‑positive rate and aggressive prompting increasing false rejections. Analyses show that this bias persists even when controlling for public corpus contamination, surface features, and literature grounding, indicating that current LLMs are not yet reliable gatekeepers for scientific rigor.
arXiv:2610.17439v1 Announce Type: new Abstract: The success of autonomous research systems relies on the quality of the scientific hypotheses they generate. Recent studies report that human reviewers...
arXiv:2606. 08251v1 Announce Type: cross Abstract: Bold projections that artificial intelligence will accelerate scientific discovery have raced ahead of evidence from working scientists, and the field still lacks large-scale, scientist-in-the-loop tests of these claims.
arXiv:2606. 07462v1 Announce Type: new Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution.
arXiv:2607. 01233v1 Announce Type: cross Abstract: LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference.
arXiv:2606. 00644v1 Announce Type: new Abstract: AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned.
ScholarCatalyst is a new benchmark that evaluates how well AI systems can retrieve research papers that inspire new work. The dataset was created by having 184 lead authors of 207 recent computer science papers annotate which earlier papers helped their projects, providing detailed rationales. The benchmark tests retrieval from the literature available at the start of a project, revealing that current agentic search and even advanced models like Claude Fable 5.1 perform only modestly better than simple embedding retrieval.