Hugging Face Trending Papers

Curbside Chatbots: Auditing the Mental Health Interventions Large Language Models Recommend to Clinicians, Caregivers, and Patients

arXiv AI
Nov 5

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

TherapyGym is a framework designed to evaluate and improve therapy chatbots on two key clinical dimensions: fidelity to evidence-based practice and safety. It uses the Cognitive Therapy Rating Scale (CTRS) to automatically score adherence to CBT techniques across multi‑turn sessions, and a multi‑label annotation scheme to assess therapy‑specific risks such as failure to address harm or abuse. The framework also introduces THERAPYJUDGEBENCH, a validation set of 116 dialogues with 1,270 expert ratings, and serves as a training harness where CTRS and safety rewards guide reinforcement learning with diverse patient simulations, leading to significant improvements in expert‑rated performance.

By Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli
arXiv AI
Oct 1

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

The paper introduces a benchmark of over 6,000 clinical triage scenarios, 7,000 physician annotations, and 225,000 large language model (LLM) responses to assess how LLMs perform under realistic variations in clinical text. The study finds that LLMs tend to recommend unnecessary care more often than physicians, especially when the input text is perturbed, and that LLM recommendations are more sensitive to gender and tone changes than human recommendations. These findings underscore the importance of deployment‑oriented evaluations that reflect expert physician behavior.

By Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi
arXiv Machine Learning
Nov 2

A prospective randomized study of patient-centered AI for clinical visit preparation

A randomized study evaluated a large language model–based chatbot designed to help patients prepare for clinical visits. In the Google Health app, 802 adults used the agent (standard or concise) or a control chatbot, and the agent outperformed the control in clinician‑rated preparation, alignment with patient context, and patient‑reported preparedness, confidence, and satisfaction. The results demonstrate that structured AI‑guided preparation can safely improve patient readiness for visits.

By Xincheng Tan, Matthew Thompson, Xiang Ji, Angelica Willis, Will Vaughan, Roma Ruparel, Michael A. Sanchez, Sean Chang, Joseph Xu, Bhavna Daryani, Preeti Singh, Pinal Bavishi, Kejia Chen, Fan Zhang, Priya Gupta, Ernest Lim, Christina Chen, Bob Lou, Dimitrios Antos, Zach Wasson, Ray Luo, Fadi Yousif, Florence Thng, Mark Malhotra, Joseph Breda, Daniel McDuff, Kavi Goel, Katherine Chou, Dale R. Webster, Avinatan Hassidim, Yossi Matias, Po-Hsuan Cameron Chen, Yun Liu, Mike Schaekermann, Quang Duong, Jackie M. Barr
arXiv AI
Oct 31

PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting

PromptSense introduces a sentiment‑aware mental health chatbot that incorporates sentiment and depression analysis with rule‑based filtering before model inference. By dynamically constructing prompts from emotional and risk insights, the system enhances contextual relevance, empathy, and safety of responses. Experiments across four model strategies and two prompt variants show that the engineered prompts outperform DSFT and RLAIF by 1.8% and 1.1% on qualitative metrics and toxicity assessments.

By Sugi Lee, Jianlong Zhou, Jing sUN
arXiv Computation and Language
Aug 31

Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires

arXiv:2510.25384v2 Announce Type: replace Abstract: Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced...

By Doan Nam Long Vu, Rui Tan, Lena Moench, Svenja Jule Francke, Daniel Woiwod, Florian Thomas-Odenthal, Sanna Stroth, Tilo Kircher, Christiane Hermann, Udo Dannlowski, Hamidreza Jamalabadi, Simone Balloccu, Shaoxiong Ji
arXiv AI
Oct 16

ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews

ReviewPilot is a conversational multi‑agent system designed to automate the PRISMA workflow for systematic literature reviews, covering both paper screening and evidence extraction. Researchers can customize screening criteria and extraction schemas through dialogue, with a project memory preserving configurations and revisions. The system records the rationale for each screening decision and the source of every extracted value, and it has demonstrated superior performance to single‑prompt LLMs and AgentSLR in epidemiology, achieving screening macro F1 scores up to 0.776 and extraction F1 up to 0.788, while also showing close agreement with human annotators in biomedicine and human‑computer interaction case studies.

By Zhaoqian Xue, Shaoze Zhou, Xuexin Li, Junting Mu, Haozheng Liu, Zhongyin Wang, Yichun He, Renkai Ma, Hao Zhou, Sarah Bratt, Siyuan Ma, Chen Chen, Lingyao Li
arXiv AI
Sep 15

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

arXiv:2609.15855v1 Announce Type: cross Abstract: % !TEX root = ../main.tex People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk con...

By Laura M. Vowels, Matthew J. Vowels, Shivali Sharma, Apoorv Jha, Rehnuma Choudhury, Wasseem El Sarraj, Rachel Francois-Walcott, Aruba Hussain, Sarah Ingram, Angela Loulopoulou, Adva Segal, Elena Volkova