arXiv Computation and Language
Leveraging a four-quadrant approach for evaluating Redpine Science
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Aug 25
K-Bench: measuring model performance on real scientific agent requests
arXiv:2608.21601v1 Announce Type: new Abstract: Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference soluti...
arXiv AI
3d ago
Sci$\Pi$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
Sciπvot is a 248‑case benchmark that evaluates how well large language model (LLM) agents perform scientific judgment during autonomous research. The benchmark is built from six months of real‑world experiments where senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly corrected the agents’ decisions. In each case, the agent must choose between its original plan and a human‑corrected alternative using only the project record available before the correction, allowing a direct comparison of LLM and human performance under varying reasoning effort and record access.
arXiv AI
Sep 1
Expert-validated STEM QA
arXiv:2608.28591v1 Announce Type: new Abstract: Recent advancements in AI are helping scientists achieve breakthroughs in fields such as mathematics, medicine, and materials sciences. New evaluation...
arXiv Computation and Language
Sep 15
Similar Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents
arXiv:2607.10198v2 Announce Type: replace Abstract: Search APIs expose ranked snippets, URLs, and metadata on which agents decide whether to answer, search again, or fetch pages. We evaluate these in...
arXiv AI
Oct 1
Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts
The paper introduces decision checkpoints that log observations and tool actions during inference, enabling the assignment of outcome categories to scientific-search agent responses. Using these checkpoints on 540 AutoResearchBench questions, the authors compare keyword search and read-first strategies, finding that keyword search yields 24.6% accuracy while raw search achieves 17.8%. The checkpoint protocol reveals detailed differences in target exposure, inspection attempts, and evidence-search calls that aggregate accuracy alone obscures.
arXiv AI
Jul 8
FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
arXiv:2607. 05682v1 Announce Type: new Abstract: LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect.