arXiv AI

HandsOnAid: Building and Evaluating an Offline First-Aid Assistant for Emergency Response Scenarios

HandsOnAid is an offline first‑aid assistant built on a 4‑bit Gemma 2B language model fine‑tuned with QLoRA on the FirstAidQA dataset. The authors evaluated seven model configurations—varying bit‑precision, safety gates, length floors, and retrieval—using a 41‑pair medically reviewed question‑answer bank and a strict 0‑5 scoring rubric with twelve safety overrides. The best performing setup, a 4‑bit fine‑tuned model with retrieval, achieved a mean score of 2.51 out of 5 and produced complete answers on a Pixel 7 Pro in under 15 seconds without network connectivity.

arXiv AI
Oct 26

ALLMTS: Automated LLM-based Medical Triage System

The paper introduces ALLMTS, an automated medical triage system that uses large language models (LLMs) to execute commercially maintained telehealth guidelines rather than directly triage callers. By selecting the appropriate guideline and evaluating its triage questions, ALLMTS deterministically derives dispositions, achieving 0.95 accuracy on 566 synthetic scenarios and 0.98 on a 50-scenario benchmark, matching expert nurse performance. The system dramatically reduces over‑triage rates (to ≤0.02) while keeping under‑triage low, demonstrating the effectiveness of coupling LLMs with executable clinical guidelines.

By Micha{\l} Remigiusz Janiszewski, Artur Stopa, {\L}ukasz Lenkiewicz, Wojciech Achtelik, Bartosz Gonczarek
arXiv AI
Oct 5

Sentence-Level Context Sensitivity as a Training-Free Detector of Unsupported Content, Evaluated Against Trained Verifiers

The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.

By Mohamed Aly Bouke
arXiv AI
Oct 31

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

The paper investigates using open‑weight large language models (LLMs) as both judges and simulated examinees to predict medical exam item difficulty from text alone. LLMs provided absolute difficulty ratings, pairwise comparisons, and response‑based signals such as correctness and option probabilities, which were combined with baseline features to improve predictions. The best models achieved Spearman correlations of .447 on USMLE and .494 on PANCE, surpassing prior shared‑task systems and matching proprietary models, demonstrating that LLM signals capture difficulty information beyond traditional text representations.

By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
arXiv AI
Jul 28

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.

By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
arXiv AI
1d ago

Screen Before You Interpret: A Portable Validity Screen for Benchmark-Based LLM Confidence Signals

The paper introduces a portable validity screen for evaluating large language model (LLM) confidence signals, mirroring clinical personality assessment practices. It computes metrics such as the rate of high‑confidence errors (L), low‑confidence correct answers (Fp), and their contrast (RBS), without adding new statistics, to determine whether a signal truly tracks correctness. Applied to 20 frontier models on 452 two‑alternative items and 33 models on 1,500 MMLU items, the screen classifies models as Valid, Indeterminate, or Invalid, revealing that many models fail to meet validity criteria depending on confidence thresholds.

By Jon-Paul Cacioli
arXiv Computation and Language
Nov 9

Safety and accuracy diverge across deployment conditions in clinical large language models

The study introduces Safety-Focused Evaluation of Scaling (SaFE-Scale) and applies it to RadSaFE-200, a 200-question radiology benchmark with clinician-defined evidence and safety labels. Thirty-four local large language models were tested under six deployment conditions, showing that providing clean clinician-written evidence significantly improved accuracy and reduced high-risk errors and dangerous overconfidence. The results indicate that safety in clinical LLMs depends more on deployment choices than on model size or inference-time compute.

By Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder, Harald K\"ostler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
arXiv Computation and Language
Oct 12

DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models

The paper introduces DistressBench, a benchmark comprising 718 English clinician-authored conversations covering 23 self-harm subcategories, each evaluated against a weighted rubric across seven crisis-support dimensions. Models are scored by the fraction of criteria they satisfy, revealing that the best models meet 88.4% of weighted criteria while clinicians achieve 98.7%. The study highlights gaps in crisis response, notably that 35.3% of conversations where crisis is recognized fail to route users to human help, underscoring the need to evaluate models on supportive care rather than mere refusal.

By Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag
arXiv AI
Aug 13

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

arXiv:2608. 12138v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings.

By Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh