arXiv AI
A primer on evaluation methods for large language models in healthcare
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
5d ago
LLJ Cards: Best practices for the Use of LLMs as Judges
arXiv:2609.24516v1 Announce Type: new Abstract: In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these s...
arXiv AI
Aug 28
Evaluating AI Generated Summaries for Cancer Patients
The study evaluates AI-generated summaries for cancer patients using a dual assessment framework that includes human experts and LLM-as-a-judge. Human domain experts—oncology clinicians and patient-facing care staff—assess summary quality on accuracy, clinical relevance, and readability. The research identifies limitations such as omissions and minor inaccuracies, which are then used to iteratively refine prompts, grounding, and safety guardrails.
arXiv AI
Jul 22
MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications
arXiv:2409. 07314v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows.
arXiv AI
Jul 14
Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences
arXiv:2607. 09932v1 Announce Type: cross Abstract: Large language models are increasingly used to summarize clinical trial results for healthcare providers, patients, and payers, but their tendency to hallucinate poses significant risks in this high-stakes context.
arXiv Computation and Language
Aug 27
Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation
The paper introduces a retrieval‑augmented multi‑agent framework that automatically generates instance‑specific evaluation rubrics for medical language models. By retrieving authoritative medical evidence, decomposing it into atomic facts, and combining these with user interaction constraints, the system produces fine‑grained criteria that outperform GPT‑4o on HealthBench and LLMEval‑Med. The generated rubrics also guide response refinement, improving medical LLM output quality by 9.2%.
arXiv AI
Jun 18
Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis
arXiv:2606. 19183v1 Announce Type: cross Abstract: Large language models (LLMs) can make clinical decision support more accessible by interpreting free-text documentation, but their direct use as diagnostic engines is limited by sensitivity to prompts, information order, and plausible but incorrect outputs.