arXiv AI

Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice

arXiv AI
Jun 9

Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs

arXiv:2606. 08483v1 Announce Type: new Abstract: Background: Consumer-facing large language models are now a common source of health information, and they interpret and personalize responses rather than retrieve them.

By Rahul Gorijavolu, Kaushik Madapati, Pritika Vig, Rawan Abulibdeh, Nikhil Jaiswal, Mahri Kadyrova, Zeamanuel Hailu Tesfaye, Charles Senteio, Paula Maurutto, Leo Anthony Celi
arXiv AI
Oct 15

Same Illness, Different Voice: Restating the Patient's Account Narrows Narrative Anchoring in Clinical Language Models

The study evaluates how clinical language models perform when patient narratives are presented in different voices—clinical, socioeconomic, and cultural—using 1,000 MedQA cases paired with three retellings each. All 12 tested models show a significant drop in accuracy (about 26% more errors) when the patient speaks in a marked voice, a penalty that persists even when the original vignette is included verbatim and is not explained by measured retelling properties. The authors find that rewriting the patient’s account into a clinical vignette before answering can halve this accuracy gap, though structured schema pipelines and additional reasoning tools offer limited improvements.

By Prabhjot Singh, Pritam Deka, Vijay Chennareddy
arXiv AI
Jun 6

Evaluating the Utility of Personal Health Records in Personalized Health AI

arXiv:2605. 18937v2 Announce Type: replace Abstract: Patient-managed Personal Health Records (PHRs) promises to empower patients to better understand their health; but information in the record is complex, potentially hindering insights.

By Rory Sayres, Kejia Chen, Ayush Jain, Matthew Thompson, Jonathan Richina, Xiang Yin, Jimmy Hu, Fan Zhang, Bob Lou, Mike Sanchez, Ines Mezerreg, Meredith Schreier, Hamsa Subramaniam, I-Ching Lee, Yugang Jia, Daniel Mcduff, Yossi Matias, Avinatan Hassidim, Dale Webster, Yun Liu, Jackie Barr, Quang Duong
arXiv AI
Sep 4

MIRA: A Bilingual Benchmark for Medical Information Response Audit

MIRA is a bilingual benchmark that evaluates whether large language models (LLMs) provide consistent medical information across different user phrasings, languages, and health literacy levels. It contains 4,320 prompts derived from 60 medically reviewed low‑risk health questions and reveals that models tend to omit key information and offer fewer concrete next steps when responding to low health‑literacy signals, a phenomenon termed Differential Information Dilution (DID). A knowledge‑guided mitigation prompt can reduce this dilution for most models, notably improving Claude and Qwen.

By Mengyu Xu, Qiaoxin Yang, Qianqian Wang, Xiwei Dai, Weiyi Wu, Chongyang Gao
arXiv Computation and Language
Aug 24

An ambiguity taxonomy for evaluating large language model performance on clinical registry abstraction: a multi-site prospective study

The study evaluates large language models (LLMs) on unprocessed electronic medical record data for clinical registry abstraction, focusing on the American College of Cardiology National Cardiovascular Data Registry. In a pilot at one academic center, the LLM identified candidate data sources for each registry question, which abstractors used to define question‑specific document sets. In a subsequent validation at a second center, the LLM answered 157 registry questions with an overall mean accuracy of 91.5%, but accuracy dropped from 96% for simple medication or event flag questions to 62% for event timing questions, reflecting increasing ambiguity and required clinical reasoning.

By James Matheson, Betsy Castillo, Andrew Y. Shin, David Scheinker
arXiv AI
Oct 2

A Multi-Agent LLM Framework for Personalized Health Checkup Interpretation and Guidance

The paper introduces a multi‑agent large language model framework that interprets personalized health checkup results by assigning distinct tasks to specialized agents and combining their outputs. Experiments on 120 Korean compound queries show that the multi‑agent approach improves overall quality scores and user preference compared to a single‑agent baseline, though it incurs higher latency and cost. The benefits are most pronounced for queries requiring personal‑record lookup.

By HyungJun Kim, Taehan Lee, Soojin Cheon