arXiv:2606. 09774v2 Announce Type: replace Abstract: Configuring an advanced scientific simulator, translating a modeling goal into a valid, runnable input deck, is a persistent bottleneck that costs domain scientists hours to days.
By Matthew Ho, Brian Liu, Jixuan Chen, Audrey Wang, Lianhui Qin
arXiv:2608. 10366v1 Announce Type: new Abstract: Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments.
By Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty, Enamul Hoque Prince
arXiv:2606. 16802v1 Announce Type: new Abstract: Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment.
By Anqi Zou, Han Deng, Chengyu Zhang, Junquan Hu, Yu Wang, Yuxiang Xing, Aokai Zhang, Hanling Zhang, Zhaoyang Liu, Ben Fei, Zhihui Wang, Wanli Ouyang
arXiv:2609.24165v1 Announce Type: new
Abstract: Synchrotron data reduction, detector calibration followed by azimuthal integration of terabyte-scale diffraction series, is a multi-step, expert-bound...
By Pawan K. Tripathi, Hemant Sharma, Andrew Chuang, Mathew J. Cherukara
FrontierChallenge is a cross‑domain benchmark that releases 300 end‑to‑end scientific workflows, of which 97 are evaluated in this study. The benchmark covers diverse fields such as quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment, and requires agents to produce a bundle of fixed scientific deliverables. Twelve frontier models were tested, and the best configurations completed only 20 of the 97 tasks, achieving a 20.6% pass rate; high partial scores and confident completion claims often did not translate into full delivery, especially in analytical chemistry and electrochemistry/environment.
By Liangcai Su, Zhaopeng Feng, Zhuo Chen, Zhen Zhang, Xiang Lin, Ruilin Li, Handuo Zhang, Ning Wang, Kailong Wen, Yueqi Guo, Feng Xing, Yiling Guo, Chenxiong Qian, Simon Shaolei Du, Lidong Bing, Xinyu Wang
ToolGate is an executable acceptance pipeline designed to streamline the creation of scientific benchmarks that rely on specialist software. It evaluates each model-generated item through three gates: (1) an executable solution script must reproduce the proposed answer, (2) a randomized no‑tool screen rejects items solvable without the software, and (3) a tool‑using agent must solve the item within a time limit. In a FEniCSx instantiation, 500 generation attempts produced 128 unique, verified benchmark items after successive filtering.
By Ke Zhang, Yankang Liu, Roya Zandi, Maziar Raissi