arXiv AI By Rezwan Islam Salvi, Saiyma Sittul Muna, Mushfiqur Rahman Mushfique

HandsOnAid: Building and Evaluating an Offline First-Aid Assistant for Emergency Response Scenarios

Read the original on arXiv AI →

HandsOnAid is an offline first‑aid assistant built on a 4‑bit Gemma 2B language model fine‑tuned with QLoRA on the FirstAidQA dataset. The authors evaluated seven model configurations—varying bit‑precision, safety gates, length floors, and retrieval—using a 41‑pair medically reviewed question‑answer bank and a strict 0‑5 scoring rubric with twelve safety overrides. The best performing setup, a 4‑bit fine‑tuned model with retrieval, achieved a mean score of 2.51 out of 5 and produced complete answers on a Pixel 7 Pro in under 15 seconds without network connectivity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 26

ALLMTS: Automated LLM-based Medical Triage System

The paper introduces ALLMTS, an automated medical triage system that uses large language models (LLMs) to execute commercially maintained telehealth guidelines rather than directly triage callers. By selecting the appropriate guideline and evaluating its triage questions, ALLMTS deterministically derives dispositions, achieving 0.95 accuracy on 566 synthetic scenarios and 0.98 on a 50-scenario benchmark, matching expert nurse performance. The system dramatically reduces over‑triage rates (to ≤0.02) while keeping under‑triage low, demonstrating the effectiveness of coupling LLMs with executable clinical guidelines.

By Micha{\l} Remigiusz Janiszewski, Artur Stopa, {\L}ukasz Lenkiewicz, Wojciech Achtelik, Bartosz Gonczarek
arXiv AI
Oct 5

Sentence-Level Context Sensitivity as a Training-Free Detector of Unsupported Content, Evaluated Against Trained Verifiers

The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.

By Mohamed Aly Bouke
arXiv AI
Oct 31

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

The paper investigates using open‑weight large language models (LLMs) as both judges and simulated examinees to predict medical exam item difficulty from text alone. LLMs provided absolute difficulty ratings, pairwise comparisons, and response‑based signals such as correctness and option probabilities, which were combined with baseline features to improve predictions. The best models achieved Spearman correlations of .447 on USMLE and .494 on PANCE, surpassing prior shared‑task systems and matching proprietary models, demonstrating that LLM signals capture difficulty information beyond traditional text representations.

By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
arXiv AI
Jul 28

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.

By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam