arXiv AI September 25, 2026 By David Gringras

IatroBench: A Pre-Registered Benchmark of Clinical Omission in Language Models

IatroBench is a pre‑registered benchmark that evaluates language models on clinical omission and commission harms across 60 scenarios and six models. Using a physician‑written rubric scored by Claude Opus 4.6, the study finds that models tend to withhold more information from patients than from doctors—a phenomenon termed framing‑contingent withholding—while also revealing varied patterns of omission across different models. The benchmark highlights how framing influences the amount of medical information shared by AI systems.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 5

IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

arXiv:2604. 07709v4 Announce Type: replace-cross Abstract: A heavily safety-trained model will hand a physician the full, patient-followable benzodiazepine taper and refuse it to the patient who needs it, over identical clinical facts; the knowledge is present either way.

By David Gringras

llms benchmarks safety

arXiv AI

Jul 22

Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety

arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.

By Koyar Afrasyab

llms multimodal benchmarks safety

arXiv AI

Jun 15

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

arXiv:2604. 14892v3 Announce Type: replace-cross Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators.

By Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor, Michael Cameron Gramanie, Ismail Kalla, Shabir A. Madhi, Kajal Morar, Marlvin T. Ncube, Haroon Saloojee, Bruce A. Bassett

llms agents benchmarks safety

arXiv AI

Jul 14

Trivial Prompt Reframing Bypasses Safety Guardrails in Google\'s MedGemma-4B

arXiv:2607. 09804v1 Announce Type: cross Abstract: Open-weight medical language models are increasingly used as the base of patient-facing and clinician-support applications.

By Avi-ad Avraam Buskila

llms multimodal benchmarks safety

arXiv AI

Sep 1

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

arXiv:2608.31016v1 Announce Type: cross Abstract: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the n...

By Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris

llms benchmarks

arXiv AI

Aug 17

Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

arXiv:2608. 14399v1 Announce Type: cross Abstract: Patients increasingly ask large language model (LLM) assistants which doctor to see, making these systems AI infomediaries: algorithms that intermediate one person's choice among other people and thereby decide, silently and at scale, which physicians become visible.

By Syeda Anshrah Gillani, Mirza Samad Ahmed Baig

llms

One email a morning, machine-written

Related stories

IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

Trivial Prompt Reframing Bypasses Safety Guardrails in Google\'s MedGemma-4B

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice