arXiv AI

Plausible Patients, Impossible Populations: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations

arXiv:2604. 17359v2 Announce Type: replace-cross Abstract: Language models asked to simulate psychiatric patients produce cases that survive inspection one at a time and populations that match no real one.

arXiv AI

Jul 22

Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety

arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.

By Koyar Afrasyab

llms multimodal benchmarks safety

arXiv AI

1d ago

Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts

arXiv:2608. 15254v1 Announce Type: new Abstract: Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI).

By Diego Mardian, Frank Liu

llms benchmarks

arXiv Machine Learning

Jul 31

KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models

arXiv:2607. 28608v1 Announce Type: new Abstract: Clinical risk models routinely achieve strong aggregate performance while producing materially different error rates across patient subgroups.

By Sparsh Roy, Samuel Girmachew, Nishita Chavan

benchmarks safety

arXiv Machine Learning

Jun 10

Quality Is Not a Safety Proxy Under Quantization

arXiv:2606. 10154v1 Announce Type: new Abstract: Quantized checkpoints are often screened first with quality metrics and only later, if at all, with direct safety tests.

By Sahil Kadadekar

llms efficiency safety

arXiv AI

Jun 16

Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering

arXiv:2606. 16890v1 Announce Type: cross Abstract: Aggregate accuracy benchmarks conceal a systematic structure in how large language models fail at electronic health record (EHR) question answering: questions requiring more inferential steps produce disproportionately more errors.

By Sanjay Basu

llms nlp benchmarks

arXiv Machine Learning

Aug 4

What EEG Foundation Models Encode: Dataset Identity and a Negative-Control Suite for Clinical Benchmarks

arXiv:2607. 24519v2 Announce Type: replace Abstract: Pretrained EEG foundation models are proposed for clinical decoding, but whether reported gains transfer across populations or survive negative controls is unclear.

By Marzieh Zare

rag fine-tuning benchmarks

arXiv AI

Jul 22

Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling

arXiv:2607. 18310v1 Announce Type: cross Abstract: Synthetic-population tools increasingly run every individual as an independent large language model (LLM) agent.

By Gurkan Ozkan

llms agents

arXiv AI

Jul 7

Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5

arXiv:2607. 04510v1 Announce Type: cross Abstract: Emergent misalignment (EM) -- the broad misbehaviour a language model acquires after fine-tuning on narrow harmful data -- is mediated in Qwen2.

By Lyndon Drake (University of Oxford), Zandi Eberstadt (University of Oxford)

llms fine-tuning

arXiv Machine Learning

Aug 4

Cross-Cohort Spectral-Temporal Dissociation in Frozen EEG Foundation-Model Representations

arXiv:2607. 24834v2 Announce Type: replace-cross Abstract: Objective.

By Marzieh Zare

rag

arXiv AI

Aug 5

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems

arXiv:2608. 03744v1 Announce Type: new Abstract: Clinical decision support is moving toward committees of language-model agents deliberating on a shared workspace.

By Sebasti\'an Andr\'es Cajas Ord\'o\~nez, Agastya Munnangi, Aldo Marzullo, Felipe Ocampo Osorio, Quang Bui, Mohammad Shahin, Armaan Grewal, Emmanuel Paul Kwesiga, Anqi Peter Li, Josephine Nanyonjo, Aaditya Panchal, Arshnoor Bhutani, Nikhil Jaiswal, Milit S. Patel, Maximin Lange, Leo Anthony Celi

llms agents benchmarks

arXiv AI

Jun 6

The Granularity Gap: A Multi-Dimensional Longitudinal Audit of Sycophancy in Gemini Models

arXiv:2606. 05183v1 Announce Type: cross Abstract: Large language models are increasingly deployed as high-stakes advisors, yet standard alignment benchmarks treat sycophancy as a binary failure mode.

By Patrick Keough

llms benchmarks safety

arXiv Machine Learning

Jun 5

IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

arXiv:2604. 07709v4 Announce Type: replace-cross Abstract: A heavily safety-trained model will hand a physician the full, patient-followable benzodiazepine taper and refuse it to the patient who needs it, over identical clinical facts; the knowledge is present either way.

By David Gringras

llms benchmarks safety

Related stories