← Back to all news
arXiv AI By Ahnaf Atef Choudhury, Ramkrishna Saha

SDoH-Aware Narrative Anchoring Bias in Medical LLMs for Trustworthy Clinical Decision Support

Read the original on arXiv AI →

The paper investigates how medical large language models (LLMs) may exhibit narrative anchoring bias when presented with the same clinical case in different patient voices. Using the NarrativeShield SDoH MedQA dataset, the authors evaluate three Qwen2.5 instruction‑tuned LLMs (1.5B, 3B, 7B) on 300 clinical cases, reporting metrics such as persona‑level accuracy, counterfactual consistency, correct consistency, and narrative sensitivity error. The 7B model achieves the highest accuracy (56.33 %) and correct consistency (40.33 %), yet narrative sensitivity errors remain substantial (31.67 %).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
4d ago

Same Illness, Different Voice: Restating the Patient's Account Narrows Narrative Anchoring in Clinical Language Models

The study evaluates how clinical language models perform when patient narratives are presented in different voices—clinical, socioeconomic, and cultural—using 1,000 MedQA cases paired with three retellings each. All 12 tested models show a significant drop in accuracy (about 26% more errors) when the patient speaks in a marked voice, a penalty that persists even when the original vignette is included verbatim and is not explained by measured retelling properties. The authors find that rewriting the patient’s account into a clinical vignette before answering can halve this accuracy gap, though structured schema pipelines and additional reasoning tools offer limited improvements.

By Prabhjot Singh, Pritam Deka, Vijay Chennareddy
llmsagents
More like this →
arXiv Computation and Language
Aug 27

MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

MTDiag is a newly released multi-turn diagnostic dialogue dataset designed to evaluate large language models (LLMs) in clinically meaningful ways. It is built from DDXPlus, MIMIC-IV, and AJCR case reports, covering both common emergency department presentations and rare conditions, and normalizes cases into a canonical schema using UMLS concept identifiers and ICD-10 codes. The dataset includes a UserLM‑8B utterance‑generation pipeline and physician‑validated natural‑language utterances, and introduces clinical knowledge‑grounded metrics that go beyond simple diagnostic accuracy for multi‑turn differential diagnosis tasks.

By Pia Chouayfati, Alexander M. Fichtl, Miriam Ansch\"utz, George Doumat, Georg Groh
llmsagentsbenchmarks
More like this →