arXiv Computation and Language

HealMed: Multilingual Evaluation of Large Language Models in Medicine

arXiv:2608. 19981v1 Announce Type: new Abstract: We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine.

arXiv AI
Sep 10

Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics

The study compared the performance of the AI system Doctorina, eight physicians, and four frontier language models on 150 synthetic Polish primary‑care consultations. Doctorina achieved 82.0% top‑1 concordance, outperforming physicians (57.0%) by 25 percentage points, and also led in diagnostic workup and treatment scores. A second run confirmed these advantages across all measured outcomes.

By Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei, Anna Kozlova, Piotr Gibas, Julian Milek, Viktar Harbachou, Aleksey Ropan, Pavel Satalkin
arXiv AI
Aug 20

Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

The paper examines how multilingual medical adaptation affects the internal representations of Whisper ASR models. By comparing various fine‑tuning strategies—zero‑shot decoding, English‑only, German‑only, two‑stage EN→EN+DE, and direct EN+DE fine‑tuning—it shows that fine‑tuning significantly improves performance, with the best model varying by setting. Layer‑wise encoder analysis reveals that English medical fine‑tuning drives the main representation shift, while multilingual continuation largely preserves the adapted space, and that domain and language signals remain recoverable across layers.

By Souranil Kahali, Rituparna Bose, Abner Hernandez, Tomas Arias-Vergara, Andreas Maier, Ning Ma, Paula Andrea Perez-Toro
Hugging Face Trending Papers
Aug 19

Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

The paper examines how multilingual medical adaptation affects the internal representations of Whisper ASR models by performing layer‑wise encoder analysis. It compares several adaptation strategies—zero‑shot decoding, English‑only fine‑tuning, German‑only diagnostic fine‑tuning, two‑stage EN→EN+DE continuation, and direct EN+DE fine‑tuning—across different Whisper sizes, finding that fine‑tuning improves performance but the best model varies by setting. Layer‑wise results show that English medical fine‑tuning drives the main encoder shift, while multilingual continuation largely preserves the adapted representation space, with domain and language information remaining recoverable across layers. "whyItMatters":"The study provides insight into how multilingual medical adaptation reshapes Whisper’s internal representations, guiding the selection of model sizes and fine‑tuning strategies for improved MedASR performance."

arXiv AI
Sep 15

A primer on evaluation methods for large language models in healthcare

arXiv:2609.14819v1 Announce Type: cross Abstract: Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and...

By Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries, Alyssa Pradhan, Timothy J Keyes, Bernardo C Bizzo, Keith J Dreyer, Sarah F Mercaldo, James M Hillis
arXiv Machine Learning
Sep 25

Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

This study benchmarks transformer models for Bangla medical named entity recognition (NER), comparing BanglaBERT, multilingual BERT (mBERT), XLM‑RoBERTa, and GPT‑4o mini under zero‑shot and few‑shot prompting. Across a full test set of 3,179 samples, fine‑tuned XLM‑RoBERTa achieves a new state‑of‑the‑art F1‑score of 0.5959, while BanglaBERT lags with 0.4937, suggesting that domain diversity outweighs language specificity. The analysis shows high performance on Medicine and Specialist entities (F1 > 0.83) but lower accuracy on Symptoms (F1 0.4367), and demonstrates that fine‑tuned transformers outperform prompt‑only approaches by a factor of 3.76.

By Rakib Abdullah, Md. Maruful Islam Maruf