TherapyGym is a framework designed to evaluate and improve therapy chatbots on two key clinical dimensions: fidelity to evidence-based practice and safety. It uses the Cognitive Therapy Rating Scale (CTRS) to automatically score adherence to CBT techniques across multi‑turn sessions, and a multi‑label annotation scheme to assess therapy‑specific risks such as failure to address harm or abuse. The framework also introduces THERAPYJUDGEBENCH, a validation set of 116 dialogues with 1,270 expert ratings, and serves as a training harness where CTRS and safety rewards guide reinforcement learning with diverse patient simulations, leading to significant improvements in expert‑rated performance.
By Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli
arXiv:2609.16590v1 Announce Type: new
Abstract: People increasingly use frontier AI models for health advice, but via different access modes (e.g., ChatGPT, ChatGPT Health, APIs) with varying setting...
By Yuan Pu, Yewon Chang, Furong Jia, Xunjian Yin, Jessica Ma, Ayman Ali, Monica Agrawal
arXiv:2610.42039v1 Announce Type: new
Abstract: Hundreds of millions of people now ask large language model (LLM) chatbots for health advice, which raises a question that is simple to pose and hard t...
By Artin Yousefi, Aizierjiang Aiersilan, Lorien C Abroms, David A Broniatowski
The paper introduces a benchmark of over 6,000 clinical triage scenarios, 7,000 physician annotations, and 225,000 large language model (LLM) responses to assess how LLMs perform under realistic variations in clinical text. The study finds that LLMs tend to recommend unnecessary care more often than physicians, especially when the input text is perturbed, and that LLM recommendations are more sensitive to gender and tone changes than human recommendations. These findings underscore the importance of deployment‑oriented evaluations that reflect expert physician behavior.
By Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi
A randomized study evaluated a large language model–based chatbot designed to help patients prepare for clinical visits. In the Google Health app, 802 adults used the agent (standard or concise) or a control chatbot, and the agent outperformed the control in clinician‑rated preparation, alignment with patient context, and patient‑reported preparedness, confidence, and satisfaction. The results demonstrate that structured AI‑guided preparation can safely improve patient readiness for visits.
By Xincheng Tan, Matthew Thompson, Xiang Ji, Angelica Willis, Will Vaughan, Roma Ruparel, Michael A. Sanchez, Sean Chang, Joseph Xu, Bhavna Daryani, Preeti Singh, Pinal Bavishi, Kejia Chen, Fan Zhang, Priya Gupta, Ernest Lim, Christina Chen, Bob Lou, Dimitrios Antos, Zach Wasson, Ray Luo, Fadi Yousif, Florence Thng, Mark Malhotra, Joseph Breda, Daniel McDuff, Kavi Goel, Katherine Chou, Dale R. Webster, Avinatan Hassidim, Yossi Matias, Po-Hsuan Cameron Chen, Yun Liu, Mike Schaekermann, Quang Duong, Jackie M. Barr
PromptSense introduces a sentiment‑aware mental health chatbot that incorporates sentiment and depression analysis with rule‑based filtering before model inference. By dynamically constructing prompts from emotional and risk insights, the system enhances contextual relevance, empathy, and safety of responses. Experiments across four model strategies and two prompt variants show that the engineered prompts outperform DSFT and RLAIF by 1.8% and 1.1% on qualitative metrics and toxicity assessments.
By Sugi Lee, Jianlong Zhou, Jing sUN
arXiv:2510.25384v2 Announce Type: replace
Abstract: Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced...
By Doan Nam Long Vu, Rui Tan, Lena Moench, Svenja Jule Francke, Daniel Woiwod, Florian Thomas-Odenthal, Sanna Stroth, Tilo Kircher, Christiane Hermann, Udo Dannlowski, Hamidreza Jamalabadi, Simone Balloccu, Shaoxiong Ji
arXiv:2609.38480v1 Announce Type: cross
Abstract: Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients presen...
By Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati, Vallijah Subasri, Caihua Shan
ReviewPilot is a conversational multi‑agent system designed to automate the PRISMA workflow for systematic literature reviews, covering both paper screening and evidence extraction. Researchers can customize screening criteria and extraction schemas through dialogue, with a project memory preserving configurations and revisions. The system records the rationale for each screening decision and the source of every extracted value, and it has demonstrated superior performance to single‑prompt LLMs and AgentSLR in epidemiology, achieving screening macro F1 scores up to 0.776 and extraction F1 up to 0.788, while also showing close agreement with human annotators in biomedicine and human‑computer interaction case studies.
By Zhaoqian Xue, Shaoze Zhou, Xuexin Li, Junting Mu, Haozheng Liu, Zhongyin Wang, Yichun He, Renkai Ma, Hao Zhou, Sarah Bratt, Siyuan Ma, Chen Chen, Lingyao Li
arXiv:2609.15855v1 Announce Type: cross
Abstract: % !TEX root = ../main.tex People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk con...
By Laura M. Vowels, Matthew J. Vowels, Shivali Sharma, Apoorv Jha, Rehnuma Choudhury, Wasseem El Sarraj, Rachel Francois-Walcott, Aruba Hussain, Sarah Ingram, Angela Loulopoulou, Adva Segal, Elena Volkova
arXiv:2607. 24754v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to provide mental health support, requiring reliable evaluation of safety, empathy, and therapeutic appropriateness.
By Asher Sprigler, Yixue Zhao, Yi Ding
arXiv:2607. 24768v1 Announce Type: new Abstract: Prenatal care is an important preventive service designed to improve outcomes for pregnant individuals.
By Vaibhav Balloli, Carissa Samuel, Samia Abdelnabi, Alex Peahl, Elizabeth Bondi-Kelly