← Back to all news
arXiv Machine Learning By Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben, Fabian Drexel, Jannik L\"ubberstedt, Johannes Moll, Anirudh Narayanan, Frederike Stumpf, Cosmin I. Bercea, Jiawen Wang, Evangelia E. Vassalou, Krischan Braitsch, Sebastian Ziegelmayer, Markus Graf, Marcus R. Makowski, Lisa Adams, Nassir Navab, Michail E. Klontzas, Keno Bressem

The Value of System 1 Models in Medicine

Read the original on arXiv Machine Learning →

The study evaluates four zero‑shot System 1 models on 86 public clinical tasks across 14 domains and 20 languages, measuring performance via AUROC. The best models achieved high AUROC scores—0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings, and 0.69 for volumetric findings—yet performed worse on imaging tasks and varied significantly with language. System 1 models were fast (0.34–0.44 s) and energy‑efficient (24–81 J) but lagged behind clinicians on triage and were outperformed by generative models in some metrics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
2d ago

Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes

The study evaluated the accuracy and calibration of several Decision and System One models on rheumatology specialty‑training exam questions and diagnostic vignettes. On the RheumaMIR exam dataset, Jev achieved 93.0% accuracy in Spanish and 91.9% in English, while GPT‑6 Sol reached 99.4%. On the RheumBench diagnostic vignettes, GPT‑6 Sol again led with 76.9% accuracy, with Jev and other System One models scoring significantly lower (50.0%–59.6%).

By Alfredo Madrid-Garc\'ia, Diego Benavent, Fabian Lechner, Phillip Kremer, Johannes Knitza, Beatriz Merino-Barbancho
llmsbenchmarks
More like this →
arXiv AI
5d ago

MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine

MedJev is an open System One model that answers medical questions in a single forward pass without generating text, enabling fast, local clinical decision support. Across ten medical benchmarks, MedJev achieves competitive accuracies—72.5 % on MedQA, 75.4 % on USMLE, and 80.1 % on NEJM—while maintaining well‑calibrated confidence estimates. On patient‑level tasks, it attains an AUROC of 0.926 for in‑hospital mortality, surpassing XGBoost, and introduces a lightweight two‑parameter recalibration method that reduces miscalibration from 0.286 to 0.007 using only 50–200 labeled examples.

By Simon A. Lee, Jiahang Sha, Moonseong Jeong, Yunbei Pan, Jeffrey N. Chiang
llmsbenchmarks
More like this →