Introducing MentalHealthBench, a new benchmark developed by OpenAI, is designed to evaluate AI responses in realistic mental health conversations. The benchmark is expert-informed, focusing on assessing both helpfulness and safety of the AI’s replies. It aims to provide a standardized way to measure performance in sensitive mental health contexts.
HealthBench is a new evaluation benchmark for AI in healthcare which evaluates models in realistic scenarios. Built with input from 250+ physicians, it aims to provide a shared standard for model performance and safety in health.
arXiv:2511.11689v4 Announce Type: replace-cross Abstract: Generative AI chatbots built for mental health could extend access to care, but evidence from real-world use is limited. We report a single-a...
arXiv:2603. 25821v2 Announce Type: replace-cross Abstract: We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions.
The paper introduces CounselReflect, a tool that converts counseling quality metrics into a framework for users to reflect on their mental‑health AI conversations. Through interviews with 21 users, the study finds that while most participants rarely reflect on their interactions, they identify specific questions they would like such a tool to address. The findings also reveal that users tend to confirm existing beliefs and focus on familiar dimensions, highlighting the need for reflection tools to expose blind spots and encourage a more comprehensive examination of AI interactions, especially when revisiting emotionally charged exchanges.
arXiv:2607. 25485v1 Announce Type: new Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf.
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
arXiv:2609.15855v1 Announce Type: cross Abstract: % !TEX root = ../main.tex People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk con...
arXiv:2507. 02983v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering.
The paper introduces a clinician‑grounded evaluation platform called InterviewPlayground, which uses a memory‑augmented patient simulator to assess AI‑assisted psychiatric intake systems. It supports comparison across different interviewing styles, reduces clinician workload, and measures clinically relevant performance. In a pilot study, a GPT‑based intake interviewer captured more relevant items but made more unfounded inferences and missed safety concerns compared to clinicians.
arXiv:2603. 25821v3 Announce Type: replace-cross Abstract: We present Doctorina MedBench, an evaluation framework for agent-based medical AI based on the simulation of physician-patient interactions.
arXiv:2607. 22692v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for emotional support despite lacking mechanisms to safely govern evolving mental health risk.
The paper surveys 61 studies on mental‑health AI and identifies a misalignment in how trust is evaluated across disciplines. It proposes a three‑layer framework—human‑oriented, interaction‑oriented, and AI‑oriented trust—and maps stakeholder perspectives onto these layers. The authors argue that future research should focus on calibrating human trust to actual interaction and AI trustworthiness rather than merely maximizing perceived trust.