The paper introduces ALLMTS, an automated medical triage system that uses large language models (LLMs) to execute commercially maintained telehealth guidelines rather than directly triage callers. By selecting the appropriate guideline and evaluating its triage questions, ALLMTS deterministically derives dispositions, achieving 0.95 accuracy on 566 synthetic scenarios and 0.98 on a 50-scenario benchmark, matching expert nurse performance. The system dramatically reduces over‑triage rates (to ≤0.02) while keeping under‑triage low, demonstrating the effectiveness of coupling LLMs with executable clinical guidelines.
By Micha{\l} Remigiusz Janiszewski, Artur Stopa, {\L}ukasz Lenkiewicz, Wojciech Achtelik, Bartosz Gonczarek
The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.
By Mohamed Aly Bouke
The paper investigates using open‑weight large language models (LLMs) as both judges and simulated examinees to predict medical exam item difficulty from text alone. LLMs provided absolute difficulty ratings, pairwise comparisons, and response‑based signals such as correctness and option probabilities, which were combined with baseline features to improve predictions. The best models achieved Spearman correlations of .447 on USMLE and .494 on PANCE, surpassing prior shared‑task systems and matching proprietary models, demonstrating that LLM signals capture difficulty information beyond traditional text representations.
By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
arXiv:2603.11413v5 Announce Type: replace-cross
Abstract: A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage p...
By David Fraile Navarro, Jialei Sheng, Farah Magrabi, Enrico Coiera
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
arXiv:2603.11413v4 Announce Type: replace-cross
Abstract: A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage p...
By David Fraile Navarro, Jialei Sheng, Farah Magrabi, Enrico Coiera
arXiv:2609.09684v2 Announce Type: replace-cross
Abstract: We study medical question-answering (QA) data selection when a developer already has questions, labels, and auxiliary explanatory texts but c...
By Yuexin Wu, Dayou Yu, Vasile Rus
The paper introduces a portable validity screen for evaluating large language model (LLM) confidence signals, mirroring clinical personality assessment practices. It computes metrics such as the rate of high‑confidence errors (L), low‑confidence correct answers (Fp), and their contrast (RBS), without adding new statistics, to determine whether a signal truly tracks correctness. Applied to 20 frontier models on 452 two‑alternative items and 33 models on 1,500 MMLU items, the screen classifies models as Valid, Indeterminate, or Invalid, revealing that many models fail to meet validity criteria depending on confidence thresholds.
By Jon-Paul Cacioli
The study introduces Safety-Focused Evaluation of Scaling (SaFE-Scale) and applies it to RadSaFE-200, a 200-question radiology benchmark with clinician-defined evidence and safety labels. Thirty-four local large language models were tested under six deployment conditions, showing that providing clean clinician-written evidence significantly improved accuracy and reduced high-risk errors and dangerous overconfidence. The results indicate that safety in clinical LLMs depends more on deployment choices than on model size or inference-time compute.
By Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder, Harald K\"ostler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
The paper introduces DistressBench, a benchmark comprising 718 English clinician-authored conversations covering 23 self-harm subcategories, each evaluated against a weighted rubric across seven crisis-support dimensions. Models are scored by the fraction of criteria they satisfy, revealing that the best models meet 88.4% of weighted criteria while clinicians achieve 98.7%. The study highlights gaps in crisis response, notably that 35.3% of conversations where crisis is recognized fail to route users to human help, underscoring the need to evaluate models on supportive care rather than mere refusal.
By Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag
arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.
By Koyar Afrasyab
arXiv:2608. 12138v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings.
By Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh