The paper investigates how rubric-based grading in medical AI benchmarks can fail when the rubrics themselves are flawed. Using the RIFT taxonomy, the authors analyze two clinical benchmarks—HealthBench Professional and LiveMedBench—and discover significant issues: nearly 30% of criteria on HealthBench Professional are non-atomic and over 65% are misaligned or rigid. They demonstrate that correcting bundled criteria can shift scores by up to 15.9 percentage points, revealing that rubric flaws materially affect evaluation outcomes.
arXiv:2610.15239v1 Announce Type: new Abstract: Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinic...
arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.
The study evaluates whether the chain-of-thought (CoT) rationales produced by medical language models truly influence their answers. Using a 30‑operator perturbation audit that modifies both the question and the CoT (e.g., severity reversal, negation flip, demographic swap, evidence ablation), the authors found that 72.9% of edits did not change the model’s answer—a high Chain‑Decoupling Rate (CDR). Across 14 large language models and four medical QA benchmarks, the CoT text had little impact on accuracy, and removing CoT prompting did not reduce performance. "whyItMatters":"The findings suggest that current medical CoT outputs may be more documentation than genuine reasoning, highlighting the need for better faithfulness checks in clinical AI systems."
Small language models can grade open‑ended exam answers as reliably as much larger models when they use an explicit rubric. In experiments with six cost‑efficient model configurations, the rubric decouples grading from judge intelligence, with answer identity explaining 95.6% of score variance and judge identity only 0.2%. Removing rubric criteria or the official answer collapses reliability and inflates scores, showing the rubric’s essential role.
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.
arXiv:2604. 07709v4 Announce Type: replace-cross Abstract: A heavily safety-trained model will hand a physician the full, patient-followable benzodiazepine taper and refuse it to the patient who needs it, over identical clinical facts; the knowledge is present either way.
The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.
arXiv:2609.01470v1 Announce Type: new Abstract: As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language...
arXiv:2603.11413v5 Announce Type: replace-cross Abstract: A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage p...
arXiv:2609.37788v1 Announce Type: cross Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, draw...
HealthBench-Psych is a mental‑health subset of the OpenAI HealthBench benchmark, created by filtering 5,000 physician‑rubric conversations for mental‑health relevance using an LLM‑applied rubric and validating the selection through two rounds of blinded clinician review. The resulting 610 conversations (12.2 % of the corpus) are released along with a pipeline, model responses, grades, and analysis code. Evaluation of 20 frontier and open models by a cross‑vendor panel of three LLM judges shows a statistically tied frontier cluster, measurable refusal behavior in two models, and near‑identical rankings across judges (τ ≥ 0.92).