A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
The Flow has not summarised this story yet — read it at arXiv AI.
The article reviews existing tools for assessing clinical reasoning in large language models, identifying six key dimensions—problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. It finds that no single instrument covers all dimensions, with some aspects reasonably addressed while others, especially faithfulness and counterfactual reasoning, remain weakly evaluated. The authors propose combining current tools through binary rubric items, separate completeness and correctness scores, importance weighting, safety caps, temporal consistency checks, and chance‑corrected reliability reporting, and call for further design work on uncertainty, counterfactual reasoning, and faithfulness in longitudinal free‑text contexts.
The paper introduces Med Evidence Bench (MEB), a benchmark of 1,536 physician‑constructed items covering guideline knowledge, standardized clinical scenarios, and real‑world laboratory scenarios, each linked to a versioned source‑text span. Across ten large language models, the Evidence Trace Score (ETS) ranged from 0.975 to 0.997 for guideline knowledge, 0.292 to 0.658 for standardized scenarios, and 0.182 to 0.447 for real‑world scenarios, with varying rankings across levels. Automated grading on 1,296 criterion‑level judgments matched the majority vote of three physicians with a macro‑F1 of 0.930, though agreement was lower for evidence application.
The paper introduces Med Evidence Bench (MEB), a benchmark of 1,536 physician-constructed items that assess clinical reasoning in large language models (LLMs) across guideline knowledge, standardized scenarios, and real‑world laboratory scenarios. Each medical claim in the benchmark is linked to a versioned source‑text span, enabling evidence traceability. Across ten LLMs, the Evidence Trace Score (ETS) varied widely by level, with high guideline recall but lower performance on real‑world scenarios and evidence application.
The paper introduces a retrieval‑augmented multi‑agent framework that automatically generates instance‑specific evaluation rubrics for medical language models. By retrieving authoritative medical evidence, decomposing it into atomic facts, and combining these with user interaction constraints, the system produces fine‑grained criteria that outperform GPT‑4o on HealthBench and LLMEval‑Med. The generated rubrics also guide response refinement, improving medical LLM output quality by 9.2%.
The paper introduces AI Consilium, a benchmark for evaluating large language models (LLMs) on providing clinically grounded, coherent, actionable, and safe second opinions in multidisciplinary settings. Using 300 NEJM–MedQA cases, the study compares eight single-model pipelines that generate role-specific opinions and interactive-consilium reports under free-form and structured conditions. Results show that structured final reports generally score higher, especially in management, and that GPT‑5.6‑Sol outperforms other models across all evaluated components, demonstrating the benchmark’s ability to differentiate model capabilities and the impact of response structure.
arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.