arXiv AI
INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 6
AutoDFT: A Closed-Loop Multi-Agent Framework for Autonomous DFT Calculations
arXiv:2605. 26179v2 Announce Type: replace-cross Abstract: Density functional theory (DFT) serves as the basis for computational discovery in materials science and chemistry, yet each calculation demands extensive human effort: adjusting algorithms when convergence stalls, revising plans when unexpected physics emerges, and inserting steps as intermediate results reshape the problem.
arXiv AI
Oct 31
Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
The paper introduces a benchmark of 250 expert-curated questions on inertial confinement fusion to evaluate language models on scientific fluency, physical reasoning, and quantitative analysis. It compares various closed‑source, open‑source, and a specialized model called Prospero, showing that Prospero improves by 10.7 percentage points and that frontier models still make critical physics errors. Retrieval assistance helps smaller models but not frontier ones, and two stress tests highlight the need for expert assessment of physical assumptions.
arXiv AI
Jul 7
Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models
arXiv:2607. 05199v1 Announce Type: new Abstract: Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows.
arXiv AI
Jul 9
AGAPI-Agents: An Open-Access Agentic AI Platform for Accelerated Materials Design on AtomGPT.org
arXiv:2512. 11935v2 Announce Type: replace Abstract: Agentic AI systems increasingly connect large language models (LLMs) to external scientific tools, yet whether and when tool access improves prediction accuracy remains uncharacterized.
arXiv AI
Jul 14
PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language
arXiv:2607. 09789v1 Announce Type: new Abstract: We introduce PHITSBench, an execution-scored benchmark for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS).
arXiv AI
Sep 18
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
SCICONVBENCH is a benchmark designed to evaluate large language models (LLMs) on multi‑turn clarification tasks in computational science. It focuses on two key abilities: eliciting missing information (disambiguation) and resolving contradictory requests (inconsistency resolution) across four domains—fluid mechanics, solid mechanics, materials science, and partial differential equations. The benchmark pairs a structured task ontology with a rubric‑based evaluation framework, measuring LLM performance in clarification behavior, conversational grounding, and final‑specification fidelity, and reveals that even top models only resolve about 52.7% of disambiguation cases in fluid mechanics while often making ungrounded assumptions.