The study evaluated the accuracy and calibration of several Decision and System One models on rheumatology specialty‑training exam questions and diagnostic vignettes. On the RheumaMIR exam dataset, Jev achieved 93.0% accuracy in Spanish and 91.9% in English, while GPT‑6 Sol reached 99.4%. On the RheumBench diagnostic vignettes, GPT‑6 Sol again led with 76.9% accuracy, with Jev and other System One models scoring significantly lower (50.0%–59.6%).
arXiv:2609.34024v1 Announce Type: cross Abstract: Jev is a non-generative "System One" model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and c...
The study evaluates Jev 1.13, a non‑generative model that selects from predefined answer options, on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena‑MCQ, and the NEJM Case Challenges. Jev’s top‑1 accuracy matches GPT‑6 Sol with medium reasoning on PubMedQA but falls behind on MetaMedQA, DiagnosisArena‑MCQ, and NEJM cases. While Jev shows strong calibration on MetaMedQA and is fast and inexpensive, its performance on examination and complex diagnostic tasks is substantially lower, indicating the need for task‑specific validation before clinical deployment.
The study evaluates four zero‑shot System 1 models on 86 public clinical tasks across 14 domains and 20 languages, measuring performance via AUROC. The best models achieved high AUROC scores—0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings, and 0.69 for volumetric findings—yet performed worse on imaging tasks and varied significantly with language. System 1 models were fast (0.34–0.44 s) and energy‑efficient (24–81 J) but lagged behind clinicians on triage and were outperformed by generative models in some metrics.
Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative la...
arXiv:2610.32478v1 Announce Type: new Abstract: Vision-language models used for decisions in histopathology rarely return calibrated probabilities over the candidates a user cares about. Decision-nat...
arXiv:2609.27607v1 Announce Type: cross Abstract: An AI-generated radiology report can resemble a physician's report while omitting an abnormality, adding an unsupported finding, or reversing its pre...
arXiv:2608. 12138v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings.
arXiv:2606. 28960v1 Announce Type: new Abstract: Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice.
The paper presents a calibrated evidence inference model fine‑tuned from Qwen3‑4B‑Base using low‑rank adapters and a classification head to classify trial outcomes as significant increase, significant decrease, or no significant difference. Trained on 1,431 open‑access PubMed Central articles, the released model (Apache 2.0) achieves an expected calibration error of 0.017, a log loss lower than Gemma 4 31B‑it in zero‑shot settings, and a macro‑F1 of 0.925 on the test split. Accepting its top‑80 % most confident predictions yields only a 2.6 % error rate, and ablation studies show that article context influences answers while the title alone still provides useful signal.
The paper presents a system for the MedReason 2026 challenge that tackles both multiple‑choice and open‑ended medical visual question answering using offline, containerized inference. Key findings include that comparing answer semantics rather than labels boosts retrieval‑only accuracy from 20.0 % to 57.5 % on a 200‑case holdout, and that varying the number of in‑prompt retrieved examples has minimal impact on final accuracy (93.5 %–94.0 %). The final system achieves 94.0 % MCQ accuracy on the development set and 93.20 % on the official pre‑evaluation, far surpassing the off‑the‑shelf baseline. "whyItMatters":"The results demonstrate that semantic‑aware retrieval and careful adapter tuning can dramatically improve medical VQA performance, offering a practical approach for high‑accuracy, offline inference in clinical settings."
arXiv:2609.13238v1 Announce Type: new Abstract: Maxillofacial report generation from cone beam computed tomography is scored here by a composite objective placing 80% of its weight on a large languag...
Fiorillo v0.5 is an open 4‑billion‑parameter model that reads a randomized trial article (cut to 6,144 tokens) and predicts whether an intervention significantly increased, decreased, or had no significant effect on an outcome. Fine‑tuned on 1,431 license‑clean training articles, it achieved an expected calibration error of 0.0168, log loss 0.8603 lower than the prior, and macro‑F1 of 0.9248 on a public test split of 1,218 prompts. The model’s release, licensed under Apache 2.0, met all four pre‑registered criteria on the Open Science Framework and was validated against the evaluated predictions.