arXiv AI
Jun 30
SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation
arXiv:2606. 30491v1 Announce Type: cross Abstract: Background.
VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise
VeriSim is a configurable patient simulation framework that injects controllable noise across six clinically grounded communication dimensions while preserving patient medical records. It uses a verifier that extracts atomic claims from utterances and checks them against a UMLS‑grounded vector index built with BioLORD embeddings, relying on structured clinical metadata rather than surface‑text similarity. Stress‑testing seven open‑weight LLMs shows realistic noise reduces diagnostic accuracy by 15‑25 percentage points and lengthens conversations by 34‑55%, with smaller models degrading more than larger ones; expert ratings confirm VeriSim’s high truth, realism, clinical utility, and noise fidelity.
Related stories
arXiv AI
Jul 29
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
arXiv:2607. 25485v1 Announce Type: new Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf.
arXiv Computation and Language
Oct 16
MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
MindEval2 is a new evaluation framework for AI chatbots in mental health support, comprising Mental Health Quality Rubrics (MHQR) and the MindSim patient simulation harness. MHQR, developed with PhD-level psychologists, provides explicit, actionable criteria for assessing response quality, while MindSim offers realistic, longitudinal patient interactions with a memory component. The study evaluated 18 models, finding that although surface-level quality has improved, guidance calibration remains the main failure across all models.
arXiv AI
Jul 28
OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
arXiv AI
Sep 25
Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
Synthetic Hospital is an open, fully synthetic longitudinal electronic health record benchmark created from public medical education material. It contains 1,268 patients and 5,602 encounters, with every diagnosis, finding, and temporal relation grounded in standard ontologies and traceable back to its source. Physicians could not reliably distinguish its records from real charts, and current frontier AI models perform significantly below human experts on tasks such as reconstructing problem lists and summarizing charts.
arXiv AI
Jul 22
Doctorina MedBench-ICD10: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI
arXiv:2603. 25821v2 Announce Type: replace-cross Abstract: We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions.
arXiv AI
Jul 9
SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care
arXiv:2601. 16529v4 Announce Type: replace Abstract: Large language models (LLMs) deployed in clinical decision support may acquiesce to patient requests for care that conflicts with evidence-based guidelines.
arXiv AI
Oct 1
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
arXiv:2609.38480v1 Announce Type: cross Abstract: Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients presen...
arXiv Computation and Language
Sep 1
Evidence-Bounded Mental Health Reasoning from Heterogeneous Speech Protocols
The paper introduces Evidence-Bounded Mental Health Reasoning, addressing the problem that current multimodal mental health screening models treat all clinical speech protocols as equally evidential. It presents the Evidence Package Benchmark, comprising 1,870 annotated packages from six diverse protocols, and proposes EviBound, a protocol-aware framework that limits reasoning to valid evidence using a planner, acoustic consensus, and a boundary critic. EviBound outperforms existing omni-modal baselines, achieving a Depression AUROC of 0.8658 with no claim violations.
arXiv AI
Jul 13
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
arXiv:2607. 09142v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice.
arXiv AI
Aug 18
DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models
arXiv:2505. 14107v5 Announce Type: replace-cross Abstract: The emergence of groundbreaking large language models capable of performing complex reasoning tasks holds significant promise for addressing various scientific challenges, including those arising in complex clinical scenarios.
arXiv AI
1d ago
MedConstBench: Benchmarking How Constitutions Steer LLMs in Medical Ethics
MedConstBench is a new medical‑ethics benchmark that treats a model’s deployment constitution as a controllable variable. It is organized around Beauchamp’s four principles and uses human‑calibrated peer review to score responses. The benchmark includes the MedDilemma dataset of clinical dilemmas and demonstrates that different constitutions influence LLM behavior in a dimension‑selective, model‑dependent way, leading to a proposed adaptive selector that chooses the best constitution per model.