DoseBench is a new benchmark consisting of 230 consumer‑style questions about whether a person can safely take another dose of acetaminophen or ibuprofen. The study evaluates eight large language models, finding that overall accuracy ranges from 44.8 % to 62.2 %, but models with similar accuracy can make vastly different dosing decisions, with some incorrectly approving up to 60.5 % of prohibited doses. The authors show that providing more guidance or drug facts can improve accuracy but may also cause models to overlook missing information, underscoring the difficulty of applying safety rules correctly in health QA.
arXiv:2606. 04262v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for everyday health questions, including whether a user can safely take another dose of an over-the-counter (OTC) medication.
arXiv:2608. 03028v1 Announce Type: new Abstract: Applying a valid medication-safety rule when its patient-specific conditions are not met can produce an incorrect decision.
FDARxBench is an expert‑curated benchmark designed to evaluate document‑grounded question answering on FDA drug label documents, focusing on generic drug assessment. It features a multi‑stage pipeline that generates high‑quality QA examples covering factual, multi‑hop, and refusal tasks, and includes protocols for both open‑book and closed‑book reasoning. Experiments with various language models show significant gaps in factual grounding, long‑context retrieval, and safe refusal behavior, highlighting the challenge of regulatory‑grade label comprehension.
The paper introduces HarmReduction, a benchmark for evaluating large language models (LLMs) on their ability to provide accurate and safe harm reduction information to people who use drugs (PWUD). The benchmark, HR-Basic, contains 2,160 question‑answer‑evidence pairs covering safety boundary checks, quantitative value provision, and polysubstance risk inference. Experiments show that even state‑of‑the‑art LLMs struggle with accuracy and can pose severe safety risks, underscoring the need for a dedicated evaluation framework.
The paper introduces a framework to predict whether a compound’s potency can be quantified in dose‑response profiling, treating quantifiability as a separate triage goal from biological activity. It shows that features from low‑cost primary screens, rather than molecular structure, strongly predict quantifiability, and that this prediction holds across new chemical scaffolds and assay families. The authors argue that incorporating quantifiability predictions can better allocate expensive dose‑response resources.
arXiv:2611.00460v1 Announce Type: cross Abstract: Large language models (LLMs) now answer medical exam questions at or above expert level, but it is unclear whether such scores reflect clinical compe...
arXiv:2610.31993v1 Announce Type: cross Abstract: AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model...
The study explores whether inexpensive three‑point primary‑screening data can predict the success of more costly 11‑point dose‑response experiments in determining compound potency (pXC50). Using a dataset of 32,971 compound‑assay pairs, random‑forest models achieved an AUROC of 0.939 and successfully identified 90% of reportable potencies while reducing follow‑up profiling to 55.7% of historically promoted pairs. The results indicate that primary‑screen response features, especially response amplitude, are more predictive of dose‑response quantifiability than molecular structure features.
arXiv:2610.18529v2 Announce Type: replace Abstract: Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To addres...
arXiv:2608. 09080v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.
The paper surveys open, non‑proprietary large language models (LLMs) and AI foundation models (AIFMs) for clinical use, presenting a taxonomy of current state‑of‑the‑art models. It empirically tests open‑weight LLMs on a personalized‑prescription task, finding that they match or surpass a proprietary baseline in overall suitability accuracy, especially when combined with Retrieval‑Augmented Generation (RAG). A subjective evaluation by blinded raters shows mixed effects of RAG on perceived suitability across multiple dimensions, and the authors discuss ethical risks and the need for responsible deployment.
arXiv:2407. 11823v4 Announce Type: replace Abstract: The United States Food and Drug Administration's (FDA's) 510(k) pathway allows manufacturers to gain medical device approval by demonstrating substantial equivalence to a legally marketed device.