arXiv AI June 16, 2026 By Sanjay Basu

Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering

arXiv:2606. 16890v1 Announce Type: cross Abstract: Aggregate accuracy benchmarks conceal a systematic structure in how large language models fail at electronic health record (EHR) question answering: questions requiring more inferential steps produce disproportionately more errors.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 28

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.

By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam

llms rag agents benchmarks

arXiv AI

6d ago

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

arXiv:2608. 12138v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings.

By Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh

llms rag benchmarks

arXiv AI

Jul 3

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.

By Samiha A. Ismail, Fan X. Chen, Ali Merali

llms benchmarks

arXiv Machine Learning

Jul 27

SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

arXiv:2605. 03534v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) grounds answers in retrieved passages, yet relevance does not guarantee sufficiency: a topical passage may still fail to justify the answer.

By Jingxi Qiu, Zeyu Han, Cheng Huang

llms rag benchmarks safety

arXiv AI

Jun 30

Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters

arXiv:2606. 30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer?

By Wenlong Wang, Fergal Reid

llms benchmarks

arXiv AI

Jun 3

AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making

arXiv:2606. 03198v1 Announce Type: cross Abstract: Clinical AI evaluation increasingly delegates scoring to large language models (LLMs) acting as AI raters, yet their scoring behavior across evaluation conditions has not been quantitatively characterized.

By Sangwon Baek, Kyu Yeon Hur, Kyunga Kim

llms

Related stories

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters

AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making