arXiv:2606. 03660v1 Announce Type: new Abstract: Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers.
By Hongyu Guo, Hao Li, He Cao, Gongbo Zhang, Li Yuan
RxnOptBench is a new benchmark that tests large language models on the task of selecting optimal reaction conditions—such as catalyst, ligand, solvent, temperature, and atmosphere—by reading real condition‑screening tables from 2025 organic‑methodology papers. The benchmark uses a continuous relative score that combines yield with stereochemical metrics (ee, dr, rr) and includes a design that separates in‑context literature use from mere memorization. Evaluation across nine frontier LLMs and three chemistry‑specialized LLMs shows that even the best models still have significant room for improvement, with chemistry‑specialized models performing at random levels on multi‑axis selection while open‑weight models approach proprietary models.
By Lingli Ge, Yubin Wang, Junyuan Gao, Jiahe Song, Jiaxing Sun, Boyu Zhu, Haote Yang, Jingchao Wang, Lixin Ma, Jiang Wu, Yuqiang Li, Conghui He
The paper introduces ChemDiffAgent, a masked diffusion language agent designed for tool‑integrated chemical reasoning. Unlike traditional autoregressive agents, it treats reasoning as iterative denoising over interleaved reasoning and tool‑use trajectories, allowing joint decision making under full‑trajectory context. A two‑stage post‑training framework further refines tool use and reasoning, and the authors benchmark the model on six chemistry tasks, reporting a ~20% performance gain over autoregressive baselines and competitive results against larger models.
By Mengdi Liu, Chenghao Jia, Hong Chang, Shiguang Shan
MolDesignBench is a new benchmark for evaluating large language model (LLM)-based agents in scenario‑grounded molecular design. It contains 2,000 generation and optimization tasks that blend implicit narrative requirements with explicit property and functional‑group constraints, including infeasible cases, and require the use of 17 specialized chemistry tools. Experiments with leading LLMs show low success rates (best ~43%) and highlight failures in implicit‑constraint reasoning, infeasibility detection, and tool usage, underscoring the benchmark’s role in identifying key bottlenecks for future research.
By Yongjun Jeong, Hanbum Ko, Ye Rin Kim, Chanhui Lee, Rodrigo Hormazabal, Jaewan Lee, Sehui Han, Sungbin Lim, Sungwoong Kim
The paper introduces MAGI, an open modular agent that orchestrates molecular‑design tools by setting objectives, launching optimization, interpreting structure–activity relationships, and adjusting strategy. In nine retrospective lead‑optimization campaigns from three pharmaceutical companies, MAGI produced valid molecules via either direct LLM generation or delegation to REINVENT 4; LLM proposals stayed closer to local chemistry and achieved comparable or higher activity with fewer operations, while REINVENT explored broader chemical space. The success of a campaign depended on the accuracy of predictive models rather than the generation route, and a blinded evaluation showed chemists could not distinguish MAGI’s outputs from expert‑generated compounds, though the SAR reasoning was deemed plausible yet incomplete. The study positions MAGI as a coordination layer that can be plugged into existing workflows, but notes that the current ceiling for real projects is set by scorer applicability rather than tool orchestration.
By Pierre Llompart, Levent Guner, Helen Lai, Alessandro Tibo, Yijie Xu
The paper introduces an LLM-as-a-Judge framework for evaluating the outputs of an agentic drug discovery assistant, ChatInvent, deployed at AstraZeneca. It defines four quality dimensions—Completeness, Relevancy, Structural Clarity, and Scope Adherence—alongside deterministic Tool Call Correctness checks, and validates the judge against five expert annotators. After optimizing the best-performing judge with few-shot demonstrations, alignment with human majority votes improves from 0.80 to 0.86, and the framework reveals that informal question phrasing does not degrade output quality.
By Emma Granqvist, Roc\'io Mercado, Samuel Genheden