arXiv AI
MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging Knowledge
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Oct 5
MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge
MS-Exam-Gen is a reproducible framework that builds a source‑grounded multiple‑choice question benchmark for evaluating large language models on knowledge about multiple sclerosis MRI. The pipeline uses expert‑indexed sources, topic induction, evidence‑grounded MCQ generation, automated quality audits, and consistency checks to produce a 3,058‑item benchmark covering 16 topics and 53 subtopics. Evaluation of 12 LLM endpoints on this benchmark revealed a wide accuracy range (89.7% to 46.9%) and identified items frequently missed by models, while audits showed reduced answer cues and position‑sensitivity in scoring.
arXiv Computation and Language
Oct 1
Jev in Medicine: A Benchmark Evaluation. Preliminary Results
arXiv:2609.34024v1 Announce Type: cross Abstract: Jev is a non-generative "System One" model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and c...
arXiv AI
Jun 9
Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model
arXiv:2606. 07721v1 Announce Type: new Abstract: Objectives: Automatic data extraction from free-text radiology reports enables large-scale research, but few studies assessed the performance of large language models (LLMs) on Dutch neuroradiology reports.
arXiv AI
Oct 2
Jev in Medicine: A Benchmark Evaluation
The study evaluates Jev 1.13, a non‑generative model that selects from predefined answer options, on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena‑MCQ, and the NEJM Case Challenges. Jev’s top‑1 accuracy matches GPT‑6 Sol with medium reasoning on PubMedQA but falls behind on MetaMedQA, DiagnosisArena‑MCQ, and NEJM cases. While Jev shows strong calibration on MetaMedQA and is fast and inexpensive, its performance on examination and complex diagnostic tasks is substantially lower, indicating the need for task‑specific validation before clinical deployment.
arXiv AI
Aug 24
An integrated diffusion-weighted imaging processing and interpretation platform for MR-guided radiotherapy
An integrated, web‑based platform has been developed to process diffusion‑weighted imaging (DWI) from MR‑guided linear accelerators and provide structured, literature‑grounded clinical interpretations. The system combines a deep‑learning pipeline for distortion correction, denoising, and IVIM/ADC fitting with a retrieval‑augmented generation (RAG) agent that references a curated knowledge base and traces each statement to its source. Independent expert ratings of nine glioblastoma cases showed high scores for clinical reasoning, citation quality, and overall utility, with a mean rating of 4.65 out of 5.
arXiv AI
Oct 15
ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail
arXiv:2610.15239v1 Announce Type: new Abstract: Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinic...