The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The study compared 13 decision models with 15 generative large language model configurations on 1,273 four‑option MedQA questions. Decision models achieved 67.71%–89.08% accuracy, while generative models ranged from 78.32%–96.86%. Mercury Decide offered the lowest cost per 1,000 questions and Clef Omni the fastest median request time, but decision models still showed higher error rates for high‑confidence predictions and varied probability repeatability.
The paper introduces three interpretable metrics—Explanation Stability Index (ESI), Causal Faithfulness Score (CFS), and Perturbation Stability Score (PSS)—to assess the faithfulness of large language models (LLMs) in clinical decision support. Evaluating six LLMs on 150 MedQA-USMLE questions, the study finds that only 23.3% of cited clinical concepts are causally necessary, and that correct answers often have lower CFS than incorrect ones, with answer consistency negatively predicting faithfulness. These findings suggest that accuracy, consistency, and consensus are incomplete safety signals for clinical decision-making support.
arXiv:2608. 09080v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.
arXiv:2603. 24481v2 Announce Type: replace Abstract: Miscalibrated confidence scores are a practical obstacle to deploying AI in clinical settings.
MedJev is an open System One model that answers medical questions in a single forward pass without generating text, enabling fast, local clinical decision support. Across ten medical benchmarks, MedJev achieves competitive accuracies—72.5 % on MedQA, 75.4 % on USMLE, and 80.1 % on NEJM—while maintaining well‑calibrated confidence estimates. On patient‑level tasks, it attains an AUROC of 0.926 for in‑hospital mortality, surpassing XGBoost, and introduces a lightweight two‑parameter recalibration method that reduces miscalibration from 0.286 to 0.007 using only 50–200 labeled examples.
OmniMed-Jev is a new medical multimodal model that represents each decision as a Choice, Noul, or Score over a runtime-supplied candidate set, returning a full probability distribution for each decision. By unifying diverse imaging modalities and prediction tasks into a single candidate-conditioned probability model, it makes heterogeneous outputs comparable probabilities rather than task-specific strings. In controlled comparisons against a generative baseline, OmniMed-Jev’s reported probabilities align more closely with observed correctness, reducing calibration error by up to an order of magnitude and reliability error by up to two, while maintaining comparable point-prediction performance.