The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2609.24965v1 Announce Type: new Abstract: Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture,...
Sciπvot is a 248‑case benchmark that evaluates how well large language model (LLM) agents perform scientific judgment during autonomous research. The benchmark is built from six months of real‑world experiments where senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly corrected the agents’ decisions. In each case, the agent must choose between its original plan and a human‑corrected alternative using only the project record available before the correction, allowing a direct comparison of LLM and human performance under varying reasoning effort and record access.
The paper investigates whether domain-specific fine‑tuning benefits open‑ended scientific reasoning in astronomy. Using a curated 300‑question QA benchmark from 2017–2026 Olympiad‑style materials, the authors compare open‑weight, API‑served general‑purpose, multimodal, and astronomy‑specialized language models. Results show that strong general‑purpose models set the highest correctness baseline, but variations in metric agreement, judge sensitivity, benchmark composition, and modality suggest that domain specialization is task‑ and deployment‑dependent and that domain‑specific evaluation is crucial for scientific workflows.
Large language models (LLMs) are increasingly used for scientific hypothesis generation. However, evaluating generated hypotheses remains a challenge for trustworthy AI-enabled scientific workflows.
arXiv:2610.34767v1 Announce Type: new Abstract: Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific...
SciGame is a new benchmark that transforms scientific literature into structured cards, each providing both an Original and a Counterfactual environment for the same task. It evaluates language‑model agents in identification and discovery scenarios under Prior‑only, Fixed, and Adaptive protocols, using historical canonical laws and a live temporal collection. Experiments with five advanced models reveal that prior knowledge can hurt performance when relations change, adaptive experimentation benefits strong models but not weak ones, and accurate explanations may mask quantitative errors, highlighting the difficulty of recovering exact symbolic laws.