HealthBench is a new evaluation benchmark for AI in healthcare which evaluates models in realistic scenarios. Built with input from 250+ physicians, it aims to provide a shared standard for model performance and safety in health.
arXiv:2511.11689v4 Announce Type: replace-cross
Abstract: Generative AI chatbots built for mental health could extend access to care, but evidence from real-world use is limited. We report a single-a...
By Thomas D. Hull, Lizhe Zhang, Caitlin A. Stamatis, Patricia A. Arean, Matteo Malgaroli
arXiv:2603. 25821v2 Announce Type: replace-cross Abstract: We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions.
By Anna Kozlova, Stanislau Salavei, Pavel Satalkin, Hanna Plotnitskaya, Sergey Parfenyuk
The paper introduces CounselReflect, a tool that converts counseling quality metrics into a framework for users to reflect on their mental‑health AI conversations. Through interviews with 21 users, the study finds that while most participants rarely reflect on their interactions, they identify specific questions they would like such a tool to address. The findings also reveal that users tend to confirm existing beliefs and focus on familiar dimensions, highlighting the need for reflection tools to expose blind spots and encourage a more comprehensive examination of AI interactions, especially when revisiting emotionally charged exchanges.
By Yahan Li, Chaohao Du, Christopher Chun Kuizon, Zeyang Li, Nimra Ishfaq, Shupeng Cheng, Angelica Yinling Sun, Adam C. Frank, Angel Hsing-Chi Hwang, Ruishan Liu
arXiv:2607. 25485v1 Announce Type: new Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf.
By Korosh Vatanparvar, Ashutosh Joshi, Maria Xenochristou, Mohammad Abuzar Hashemi, Prasad Kasu, Deepak Bansal, Daniel Lopez-Martinez, Anchal Nema, Ramya Ganesan, Will Kimbrough, Alex Woody, Yadunandana Rao, Dilek Hakkani-Tur, Wilko Schulz-Mahlendorf
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam