arXiv AI
A primer on evaluation methods for large language models in healthcare
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 22
LLJ Cards: Best practices for the Use of LLMs as Judges
arXiv:2609.24516v1 Announce Type: new Abstract: In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these s...
arXiv AI
Aug 28
Evaluating AI Generated Summaries for Cancer Patients
The study evaluates AI-generated summaries for cancer patients using a dual assessment framework that includes human experts and LLM-as-a-judge. Human domain experts—oncology clinicians and patient-facing care staff—assess summary quality on accuracy, clinical relevance, and readability. The research identifies limitations such as omissions and minor inaccuracies, which are then used to iteratively refine prompts, grounding, and safety guardrails.
arXiv AI
Sep 30
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
arXiv:2609.37788v1 Announce Type: cross Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, draw...
arXiv AI
Jul 22
MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications
arXiv:2409. 07314v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows.
arXiv Computation and Language
2d ago
Safety and accuracy diverge across deployment conditions in clinical large language models
The study introduces Safety-Focused Evaluation of Scaling (SaFE-Scale) and applies it to RadSaFE-200, a 200-question radiology benchmark with clinician-defined evidence and safety labels. Thirty-four local large language models were tested under six deployment conditions, showing that providing clean clinician-written evidence significantly improved accuracy and reduced high-risk errors and dangerous overconfidence. The results indicate that safety in clinical LLMs depends more on deployment choices than on model size or inference-time compute.
arXiv AI
Jul 14
Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences
arXiv:2607. 09932v1 Announce Type: cross Abstract: Large language models are increasingly used to summarize clinical trial results for healthcare providers, patients, and payers, but their tendency to hallucinate poses significant risks in this high-stakes context.