← Back to all news
arXiv AI By Rotem Gershon, Seffi Cohen, Dean Ariel, Havatzelet Bilavsky-Yarden, Amiel Sberro, Shai Yitzhaki, Osnat Tausky, Naama Golan, Maayan Rabinovitch, Ariel Avraham Hasidim, Itay Arianu, Maya Dagan, Ran Balicer, Noa Dagan

Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

Read the original on arXiv AI →

The article reports on a study comparing Jev, a decision language model that provides structured judgments, with large language model (LLM) judges for evaluating clinical AI outputs. Across multiple use cases—including pediatric answer quality, binary HealthBench criteria, MedHELM open‑ended tasks, and HealthBench Professional—Jev matched or exceeded the performance of several frontier and open‑weight LLM judges. Moreover, Jev was significantly cheaper and faster, costing 73–294 times less and taking 7–57 times less time than frontier judges, suggesting it could enable efficient, scalable clinical AI evaluation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 15

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

arXiv:2604. 14892v3 Announce Type: replace-cross Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators.

By Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor, Michael Cameron Gramanie, Ismail Kalla, Shabir A. Madhi, Kajal Morar, Marlvin T. Ncube, Haroon Saloojee, Bruce A. Bassett
llmsagentsbenchmarkssafety
More like this →
arXiv AI
Oct 16

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

The paper introduces AI Consilium, a benchmark for evaluating large language models (LLMs) on providing clinically grounded, coherent, actionable, and safe second opinions in multidisciplinary settings. Using 300 NEJM–MedQA cases, the study compares eight single-model pipelines that generate role-specific opinions and interactive-consilium reports under free-form and structured conditions. Results show that structured final reports generally score higher, especially in management, and that GPT‑5.6‑Sol outperforms other models across all evaluated components, demonstrating the benchmark’s ability to differentiate model capabilities and the impact of response structure.

By Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev, Mikhail Yu. Valkov, Valeria Yu. Chernina, Natalia Gombolevskaya, Ayyaz Hussain, Vladimir M. Agafonov, Victor A. Gombolevskiy, Aleksandr V. Nesterov, Iaroslav R. Bespalov, Ivan V. Oseledets, Dmitry V. Dylov
llmsbenchmarks
More like this →
arXiv AI
Aug 13

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

arXiv:2608. 12138v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings.

By Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh
llmsragbenchmarks
More like this →
arXiv AI
Jul 28

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.

By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
llmsragagentsbenchmarks
More like this →