SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

SWE‑bench Science is a benchmark for evaluating coding agents on scientific software engineering tasks, comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. The tasks are grouped into Issue‑driven, Expert‑exploratory, and Engineering‑integration paradigms, and even the best agent, Claude Code with Opus‑5 (max), achieves a pass@1 below 50%. The study identifies four common failure mechanisms—lack of scientific knowledge, misguided exploration, incomplete repair coverage, and poor generalization—and shows that providing scientific guidance can both help and hinder repair depending on its alignment with the task.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

• llms
• agents
• benchmarks