← Back to all news
arXiv AI By Betty Xiong, Jillian Fisher, Benjamin Newman, Meng Hu, Shivangi Gupta, Yejin Choi, Lanyan Fang, Russ B Altman

FDARxBench: Benchmarking Regulatory and Clinical Reasoning on FDA Generic Drug Assessment

Read the original on arXiv AI →

FDARxBench is an expert‑curated benchmark designed to evaluate document‑grounded question answering on FDA drug label documents, focusing on generic drug assessment. It features a multi‑stage pipeline that generates high‑quality QA examples covering factual, multi‑hop, and refusal tasks, and includes protocols for both open‑book and closed‑book reasoning. Experiments with various language models show significant gaps in factual grounding, long‑context retrieval, and safe refusal behavior, highlighting the challenge of regulatory‑grade label comprehension.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 31

Can I Take Another Dose? DoseBench: Evaluating LLM Decision-Making in OTC Dosing QA

DoseBench is a new benchmark consisting of 230 consumer‑style questions about whether a person can safely take another dose of acetaminophen or ibuprofen. The study evaluates eight large language models, finding that overall accuracy ranges from 44.8 % to 62.2 %, but models with similar accuracy can make vastly different dosing decisions, with some incorrectly approving up to 60.5 % of prohibited doses. The authors show that providing more guidance or drug facts can improve accuracy but may also cause models to overlook missing information, underscoring the difficulty of applying safety rules correctly in health QA.

By Maroof Kousar, Jinsung Han, Mohit Chandra, Munmun De Choudhury, Yibo Hu
llmsbenchmarkssafety
More like this →
arXiv Computation and Language
Aug 24

MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation

MedRAGChecker is a claim-level verification framework designed for biomedical retrieval‑augmented generation (RAG). It decomposes generated answers into atomic claims and assesses each claim’s support by combining evidence‑grounded natural language inference with biomedical knowledge‑graph consistency signals. The aggregated claim decisions provide diagnostics that distinguish retrieval and generation failures, such as faithfulness, under‑evidence, contradiction, and safety‑critical errors, and the system is distilled into compact models for scalable evaluation.

By Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang
llmsragbenchmarkssafety
More like this →