arXiv AI
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 28
OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
arXiv AI
Jul 29
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
arXiv:2607. 25485v1 Announce Type: new Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf.
arXiv AI
Aug 14
Doctorina MedBench: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI
arXiv:2603. 25821v3 Announce Type: replace-cross Abstract: We present Doctorina MedBench, an evaluation framework for agent-based medical AI based on the simulation of physician-patient interactions.
arXiv AI
Jul 22
Doctorina MedBench-ICD10: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI
arXiv:2603. 25821v2 Announce Type: replace-cross Abstract: We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions.
arXiv Computation and Language
Aug 27
MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation
MTDiag is a newly released multi-turn diagnostic dialogue dataset designed to evaluate large language models (LLMs) in clinically meaningful ways. It is built from DDXPlus, MIMIC-IV, and AJCR case reports, covering both common emergency department presentations and rare conditions, and normalizes cases into a canonical schema using UMLS concept identifiers and ICD-10 codes. The dataset includes a UserLM‑8B utterance‑generation pipeline and physician‑validated natural‑language utterances, and introduces clinical knowledge‑grounded metrics that go beyond simple diagnostic accuracy for multi‑turn differential diagnosis tasks.
arXiv AI
Jul 10
MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
arXiv:2607. 08257v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet existing benchmarks rarely simulate complete psychiatric clinical encounters.