arXiv Computation and Language
Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Sep 18
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
oMeBench is a large-scale, expert-curated benchmark designed to evaluate large language models (LLMs) on organic mechanism reasoning. It contains over 10,000 annotated mechanistic steps, including reaction type labels, intermediate structures, and difficulty ratings, and introduces the oMeS scoring framework to assess logical consistency and chemical structural similarity. Evaluation shows that while current LLMs display promising chemical intuition, they often fail to produce correct and consistent multi-step reasoning, though prompting and fine-tuning can bring smaller models up to the level of closed‑source frontier models.
arXiv Machine Learning
Jun 5
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
arXiv:2606. 05693v1 Announce Type: new Abstract: Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained.
arXiv AI
Jun 3
From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
arXiv:2606. 03660v1 Announce Type: new Abstract: Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers.
arXiv AI
5d ago
ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning
arXiv:2610.17183v1 Announce Type: new Abstract: Current chemistry benchmarks primarily evaluate outcome prediction or reaction pathway generation, but rarely test whether a model can explain why an o...
arXiv AI
Sep 1
CoMPASS: Collaborative Molecular Property Prediction via Adaptive Small-Large Model Synergy
arXiv:2608.30674v1 Announce Type: cross Abstract: Accurate molecular property prediction requires both statistical reliability and chemical reasoning. Graph neural networks can be calibrated directly...
arXiv Machine Learning
Sep 22
Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks
The paper introduces OmicsBench, a new reasoning benchmark for multi‑omics sequences that includes 1,160 expert‑validated questions across DNA regulation, RNA processing, and protein function tasks, requiring traceable evidence chains. Evaluation of 17 large language models shows that scientific LLMs, while more accurate in classification, often lack valid evidence, suggesting shortcut learning. To address this, the authors propose tool‑augmented on‑policy distillation (TA‑OPD), a post‑training method that improves both evidence grounding and predictive performance across five Qwen3.5 models of varying sizes.