arXiv AI August 13, 2026 By Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

Read the original on arXiv AI →

arXiv:2608. 12138v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 28

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.

By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam

llms rag agents benchmarks

arXiv AI

1d ago

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

arXiv:2608. 16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation.

By Yifan Zhang, Rahmatollah Beheshti

llms nlp benchmarks safety

arXiv AI

Jul 3

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.

By Samiha A. Ismail, Fan X. Chen, Ali Merali

llms benchmarks

arXiv AI

Jun 24

MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

arXiv:2606. 24200v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora.

By Junhyeok Lee, Han Jang, Hyeonjin Goh, Kyu Sung Choi

rag benchmarks safety

Hugging Face Trending Papers

2d ago

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart.

llms nlp benchmarks safety

arXiv AI

Jun 30

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

arXiv:2606. 28960v1 Announce Type: new Abstract: Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice.

By Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena

llms benchmarks

Related stories

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries