arXiv:2607. 19104v1 Announce Type: cross Abstract: Large language models (LLMs) excel at general-purpose code generation, yet how well they handle scientific code remains an open question.
By Weifeng Sun, Ye Fan, Yuchen Chen, Gou Tan, Jieke Shi, Yuan Yidi, Swee Liang Wong, Jonathan Pan, David Lo
arXiv:2608. 19799v1 Announce Type: new Abstract: Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions.
By Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
By Tianyu Ding, Aditya Nannapaneni, Bingfan Liu, Ling Zhang
arXiv:2609.00654v1 Announce Type: new
Abstract: We describe the SciTrue team's participation in both subtasks of the NTCIR-19 SciClaimEval task~\cite{sciclaimeval}, which asks systems to verify scien...
By Qiming Bao, Ne\c{s}et \"Ozkan Tan, Siyuan Wang, Mark Gahegan
SWE‑bench Science is a benchmark for evaluating coding agents on scientific software engineering tasks, comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. The tasks are grouped into Issue‑driven, Expert‑exploratory, and Engineering‑integration paradigms, and even the best agent, Claude Code with Opus‑5 (max), achieves a pass@1 below 50%. The study identifies four common failure mechanisms—lack of scientific knowledge, misguided exploration, incomplete repair coverage, and poor generalization—and shows that providing scientific guidance can both help and hinder repair depending on its alignment with the task.
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman