PhysioBench is a unified benchmark for physiological signal question answering that consolidates annotations from 22 public datasets into 61.4 million question‑answer pairs spanning 30 tasks. Each pair is linked to a specific signal segment and traceable to its source annotation. The benchmark evaluates 21 models—including large language models, vision‑language models, time‑series language models, and physiological signal foundation models—under three settings, revealing that no model consistently excels across all modalities and tasks, and that performance is sensitive to question wording.
arXiv:2605. 29483v2 Announce Type: replace Abstract: Wearable devices enable continuous monitoring of physiological signals such as ECG and PPG, but existing mHealth systems are largely limited to task-specific prediction pipelines or reactive question answering over static summaries.
WearableQA is a new benchmark that tests AI systems on health reasoning using real-world wearable data from 200 users, each with up to 500 days of daily measurements. It contains 4,084 ten‑option multiple‑choice questions derived from wearable time series, blood biomarkers, and demographics, and is organized into 16 question types that distinguish data‑driven computation from physiological interpretation and single‑signal from cross‑signal reasoning. Evaluation of 14 large language models shows wide performance gaps, indicating that the benchmark remains challenging and useful for diagnosing model capabilities.
MMTClinic is a new benchmark that tests large language models on complex reasoning and question‑answering tasks involving clinical time‑series data. It combines text, medical images, and multivariate physiological signals to create 30,000 QA pairs—including 15,000 multiple‑choice and 15,000 open‑ended questions—in five languages (English, Hindi, Bengali, Marathi, and Tamil). The benchmark covers mortality prediction, heart‑rate forecasting, and SOFA score estimation, and evaluates 13 state‑of‑the‑art LLMs across zero‑shot, few‑shot, and chain‑of‑thought settings, revealing significant performance gaps across tasks, languages, and modalities.
ECGQuest is a new benchmark that evaluates language models on the contextual knowledge required for electrocardiogram interpretation, featuring 10,904 True/False questions derived from 23 ECG references and 2003‑2025 Computing in Cardiology proceedings. The study tested 23 commercial and open‑source models, finding that zero‑shot accuracy ranged from 49.5% to 74.4% and that fine‑tuning with Low‑Rank Adaptation improved all open‑source models by 6.5–14.1%, with the best fine‑tuned model achieving 76.3% accuracy and a five‑model ensemble reaching 78.5%. ECGQuest demonstrates that parameter‑efficient fine‑tuning can enable smaller models to compete with larger commercial ones on ECG‑specific tasks.
arXiv:2608. 20315v1 Announce Type: new Abstract: Predictive models over structured electronic health records (EHRs) remain central to machine learning for healthcare, but few have jointly emphasized quantitative laboratory information and interpretability with respect to input medical events.
The paper introduces BRIE, a continuously maintainable benchmark for evaluating large language models (LLMs) in electronic health record (EHR) information retrieval. It presents a scalable framework that automatically generates question–answer pairs from longitudinal EHR notes, validated by nineteen clinicians. The benchmark allows assessment of multiple inference strategies and highlights that state‑of‑the‑art LLMs often miss clinically important information, especially when synthesis across documents is required.
The paper introduces BRIE, a scalable framework that automatically creates question–answer pairs from longitudinal electronic health record notes, validated by nineteen clinicians. It offers a continuously maintainable benchmark for evaluating large language models in clinical settings, addressing limitations of manual, costly, and quickly outdated existing benchmarks. Experiments across nine LLMs and five inference strategies reveal that even state‑of‑the‑art systems often miss clinically important information, especially for synthesis‑heavy queries.
arXiv:2608.24327v2 Announce Type: replace Abstract: With the advent of Large Language Models and its instruction following capabilities a promising application is the task of summarization. Within th...
arXiv:2603. 06638v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has shifted time series analysis from narrow analytics to general-purpose reasoning.
arXiv:2607. 09880v1 Announce Type: cross Abstract: Clinical time series are central to patient monitoring, risk assessment, and clinical decision support.
arXiv:2607. 25947v1 Announce Type: new Abstract: Question answering (QA) over irregular clinical time series (ICTS) plays a pivotal role in a wide range of healthcare applications.
arXiv:2606. 02802v1 Announce Type: new Abstract: Large language models (LLMs) exhibit strong natural-language reasoning abilities for clinical decision support, but struggle to effectively model structured longitudinal electronic health records (EHRs).