← Back to all news
arXiv AI By Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens, Asma Ben Abacha

Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies

Read the original on arXiv AI →

The paper evaluates large language model (LLM) judges for radiology report assessment across multiple imaging modalities and anatomical regions, comparing four existing metrics (RadFact, GREEN, FineRadScore, CRIMSON) with a new metric called VERT. GREEN emerges as the most robust among current metrics, while VERT improves correlation with radiologist judgments by up to 26% over GREEN. The study also shows that ensembling and lightweight fine‑tuning of the Qwen3‑30B model can yield additional performance gains and significant latency reductions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Jul 30

Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance

arXiv:2607. 26333v1 Announce Type: cross Abstract: Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment.

By Panagiotis Fytas, Ian Selby, Clemens Karner, Judith Babar, Simon Baker, Jake Beckford, Timothy J. Sadler, Shahab Shahipasand, Arthikkaa Thavakumar, John Li Chen, Alex Sawer, Michael Roberts, Jonathan Weir-McCall, J. H. F. Rudd, Carola-Bibiane Sch\"onlieb, Anna Korhonen, Anna Breger
llmsfine-tuningmultimodalsafety
More like this →
arXiv AI
Sep 17

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

The study examines how differences in radiologists’ reporting styles—such as terminology, shorthand, formatting, and detail—affect the evaluation of AI-generated chest X‑ray reports. By quantifying the sensitivity of common metrics to these variations, the authors show that changes in reference reports can shift model rankings. They introduce a taxonomy of reporting variations and a rewriting method, ReRef, that preserves clinical meaning while altering style, and release a validated dataset of paired reference reports to aid future research.

By Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny, Nitya M. Bhalla, Fatma Uyar Morency, Pradeep Ravikumar, Zachary C. Lipton, Michael Oberst
More like this →
arXiv Computer Vision
Sep 22

Performance vs Consistency: Evaluating a Foundation Model in Lung-RADS Screening

arXiv:2609.22281v1 Announce Type: new Abstract: Foundation models have recently demonstrated strong capabilities across a wide range of medical imaging tasks. However, their performance in structured...

By Benjamin Renoust, Pierre Baudot, Tiffany Foriel, Yousra Haddou, Charles Voyton, Pierre-Henri Siot, Ezequiel Geremia, Danny Francis, Jean-Christophe Brisset, Val\'erie Bourd\`es, Sylvain Bodard, Benoit Huet
llmsfine-tuning
More like this →