arXiv Computation and Language

LLMs Anchor on Chief Complaint and Fail to Integrate Evidence in Sequential Clinical Triage

The study evaluates large language models (LLMs) on sequential emergency department triage, where acuity labels are predicted from progressively longer nurse‑patient conversations. Six LLMs were tested at five checkpoints on simulated and physician‑authored dialogues, showing a decline from moderate‑to‑substantial agreement on full records to only fair‑to‑moderate agreement at each checkpoint. The models consistently anchor on chief complaint exchanges and fail to integrate later evidence, yielding low agreement with clinicians (QWK 0.295 vs. 0.887‑0.929) and concentrating predictions on ESI‑2 and ESI‑3. whyItMatters":"The findings reveal that LLMs, despite strong offline performance, cannot reliably handle the sequential nature of real‑time triage, highlighting a critical gap for safe deployment in emergency settings."

arXiv AI
Aug 3

Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support

arXiv:2607. 28677v1 Announce Type: new Abstract: LLM now pass medical licensing examinations and, in curated cases, can rival physicians at diagnostic reasoning.

By Shayndhan Sivanathan, Shravan Nageswaran, Mehdi Zadem, Ryaan Sultan, Nicolas von Mallinckrodt, Max Solovyev, Alexey Matyushkin, Sumon Sadhu, Gabriele C DeLuca, Sanjeeva Jeyaretna, James Hillis, Manoj Ramachandran, Prakash Jayakumar
arXiv AI
Jul 16

Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry

arXiv:2607. 13036v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for decision support in healthcare, but clinical evidence is often incomplete or evolving.

By Oriana Presacan, Andreea Grama, Larisa Irimin\u{a}, Alireza Nik, Jaya Ojha, Vajira Thambawita, Ciprian I. B\u{a}cil\u{a}, Bogdan Ionescu, Michael A. Riegler
arXiv Computation and Language
Nov 9

Safety and accuracy diverge across deployment conditions in clinical large language models

The study introduces Safety-Focused Evaluation of Scaling (SaFE-Scale) and applies it to RadSaFE-200, a 200-question radiology benchmark with clinician-defined evidence and safety labels. Thirty-four local large language models were tested under six deployment conditions, showing that providing clean clinician-written evidence significantly improved accuracy and reduced high-risk errors and dangerous overconfidence. The results indicate that safety in clinical LLMs depends more on deployment choices than on model size or inference-time compute.

By Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder, Harald K\"ostler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
arXiv AI
Oct 26

ALLMTS: Automated LLM-based Medical Triage System

The paper introduces ALLMTS, an automated medical triage system that uses large language models (LLMs) to execute commercially maintained telehealth guidelines rather than directly triage callers. By selecting the appropriate guideline and evaluating its triage questions, ALLMTS deterministically derives dispositions, achieving 0.95 accuracy on 566 synthetic scenarios and 0.98 on a 50-scenario benchmark, matching expert nurse performance. The system dramatically reduces over‑triage rates (to ≤0.02) while keeping under‑triage low, demonstrating the effectiveness of coupling LLMs with executable clinical guidelines.

By Micha{\l} Remigiusz Janiszewski, Artur Stopa, {\L}ukasz Lenkiewicz, Wojciech Achtelik, Bartosz Gonczarek
arXiv AI
Oct 2

Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage

The study evaluates counterfactual bias in ten open‑source large language models (LLMs) for pediatric Emergency Severity Index (ESI) prediction. By creating paired clinical vignettes that differ only in demographic or socioeconomic variables, the authors measure shifts in acuity assignment, finding that counterfactual sensitivity varies widely across model families and sizes. A fine‑tuned Qwen2.5‑7B model exhibited the lowest sensitivity, while larger or medical‑domain models sometimes showed greater shifts, highlighting the need for fairness assessment before clinical deployment.

By Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang
arXiv AI
Oct 5

Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves

Large language models (LLMs) are increasingly used for clinical reasoning, yet their ability to revise judgments as patient evidence changes is uncertain. In this study, researchers evaluated longitudinal belief updating using real intensive‑care patient trajectories and found that conditioning on prior judgments often increased prediction error. Controlled experiments revealed two failure modes: models were more responsive to worsening than improving respiratory evidence, and they shifted estimates significantly when prior risk was altered, indicating a causal influence of prior beliefs. Prompting did not improve reliability, and an Evidence‑Validated Longitudinal Update (EVLU) approach produced fewer but more trustworthy revisions, highlighting a reliability‑coverage trade‑off.

By Min Zeng, Rui Zhang
arXiv AI
Oct 16

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

The paper introduces a benchmark of 400 synthetic healthcare workflow episodes to evaluate large language model (LLM) agents’ reliability in escalating care when evidence is missing or conflicting. It finds that while clean decision accuracy is high (0.97), models miss a large proportion of necessary escalations (32–97%) under uncertainty, and even the largest 9.7B model’s performance drops to 0.78 in conflict scenarios. A workflow-aware controller improves utility and reduces missed escalations, but its effectiveness is limited by lexical generalization failures.

By Zihao Zheng (Washington University in St. Louis), Junyi Yao (Washington University in St. Louis), Baichuan Li (Southern Methodist University), Jiayu Long (Washington University in St. Louis)
arXiv Computation and Language
Oct 12

DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models

The paper introduces DistressBench, a benchmark comprising 718 English clinician-authored conversations covering 23 self-harm subcategories, each evaluated against a weighted rubric across seven crisis-support dimensions. Models are scored by the fraction of criteria they satisfy, revealing that the best models meet 88.4% of weighted criteria while clinicians achieve 98.7%. The study highlights gaps in crisis response, notably that 35.3% of conversations where crisis is recognized fail to route users to human help, underscoring the need to evaluate models on supportive care rather than mere refusal.

By Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag