arXiv AI
Jun 12
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
arXiv:2606. 13020v1 Announce Type: new Abstract: Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction.
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
SCICONVBENCH is a benchmark designed to evaluate large language models (LLMs) on multi‑turn clarification tasks in computational science. It focuses on two key abilities: eliciting missing information (disambiguation) and resolving contradictory requests (inconsistency resolution) across four domains—fluid mechanics, solid mechanics, materials science, and partial differential equations. The benchmark pairs a structured task ontology with a rubric‑based evaluation framework, measuring LLM performance in clarification behavior, conversational grounding, and final‑specification fidelity, and reveals that even top models only resolve about 52.7% of disambiguation cases in fluid mechanics while often making ungrounded assumptions.
Related stories
arXiv AI
Jun 3
X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes
arXiv:2603. 05290v2 Announce Type: replace Abstract: Large language models (LLMs) achieve promising performance, yet their ability to reason remains poorly understood.
arXiv AI
6d ago
Re4: Scientific Computing Agent with Rewriting, Resolution, Review and Revision
The paper introduces Re4, a scientific computing agent that employs a four-step logical chain—rewriting, resolution, review, and revision—using three large language models: a Consultant, a Programmer, and a Reviewer. The Consultant rewrites problem descriptions by linking them to domain knowledge, the Programmer generates and runs code, and the Reviewer iteratively refines the code through self-debugging. Experiments on PDEs, fluid dynamics, ill‑conditioned linear systems, and data‑driven physics show that this collaborative framework markedly improves bug‑free code generation and execution success rates across several LLMs.
arXiv AI
Jul 9
MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
arXiv:2607. 07391v1 Announce Type: new Abstract: Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue.
arXiv AI
Aug 5
Large language models for partial differential equation workflows
arXiv:2608. 03600v1 Announce Type: new Abstract: Partial differential equations (PDEs) become actionable in science and engineering not as isolated formulae, but as executable workflows that connect modelling assumptions, governing equations, numerical solvers, diagnostics, and decisions.
arXiv AI
Jun 9
MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
arXiv:2510. 12171v2 Announce Type: replace Abstract: Large Language Models have shown strong scientific reasoning ability, but their performance on materials science problems remains less studied.
arXiv AI
3d ago
Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
The paper introduces a benchmark of 250 expert-curated questions on inertial confinement fusion to evaluate language models on scientific fluency, physical reasoning, and quantitative analysis. It compares various closed‑source, open‑source, and a specialized model called Prospero, showing that Prospero improves by 10.7 percentage points and that frontier models still make critical physics errors. Retrieval assistance helps smaller models but not frontier ones, and two stress tests highlight the need for expert assessment of physical assumptions.
arXiv AI
3d ago
PAPERBENCHX: Can AI Reproduce Science Across Disciplines?
PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.
arXiv AI
Jul 24
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
arXiv:2607. 20926v1 Announce Type: new Abstract: Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources.
arXiv Computation and Language
Sep 1
SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks
arXiv:2606.15872v2 Announce Type: replace Abstract: Frontier scientific reasoning remains a major challenge for large language models (LLMs), where even the strongest commercial systems fall short of...
arXiv Machine Learning
Aug 27
SciMIF: Understanding Multimodal Instruction Following in Scientific Domains
SciMIF is a new benchmark that evaluates how well multimodal large language models (MLLMs) can follow complex scientific instructions. It is built on an analysis of 22 tasks across five scientific fields and introduces a taxonomy of 10 constraint groups that capture both general and discipline‑specific requirements. Experiments show large performance gaps between fields—chemistry is hardest—and that larger models do not necessarily improve constraint adherence, especially for fine‑grained, knowledge‑heavy instructions.
arXiv AI
Oct 2
Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI
The paper introduces a pipeline that automatically creates ontology‑grounded multiple‑choice question benchmarks for evaluating large language models (LLMs) on logical reasoning tasks in scientific AI. By using OWL 2 ontologies, correct answers are guaranteed by design and distractors are generated and formally verified as incorrect through an OWL reasoner. Experiments on three ontologies—Pizza, PMDco, and DOID—yielded 112, 2,491, and 15,216 MCQs, respectively, with high natural‑language quality and challenging zero‑shot performance for six LLMs.