arXiv AI

Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Text

arXiv Computation and Language
Oct 23

From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries

The paper explores how Large Language Models can translate structured clinical data into patient-friendly narratives while maintaining clinical accuracy. Using a constrained generation framework that blends evidence conditioning with Retrieval-Augmented Generation, the authors evaluate alignment through Natural Language Inference, factuality metrics, and readability. Their results show that grounding constraints improve clinical alignment and uncertainty preservation across three LLMs, though they may increase reading difficulty and medical terminology.

By Bayram Ayadi, Florian Geissler, Johanna Schmidhuber
Hugging Face Trending Papers
Jun 22

The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models

Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods.

arXiv AI
Aug 28

Making Clinical Language Models Auditable: Concept-Guided Fine-Tuning for Robust Prediction

The paper introduces CAST, a concept-guided artifact suppression tuning framework that uses sparse autoencoders to identify and suppress note-specific artifacts in clinical language models. CAST labels latent features with an LLM-assisted pipeline and ICD‑10 constraints, then fine‑tunes the model while providing post‑hoc per‑concept attributions for auditability. In experiments on MIMIC‑IV discharge‑note mortality prediction, CAST outperforms standard fine‑tuned encoders and competes with strong LLM baselines while offering a feature‑level audit trail of clinical concepts and suppressed artifacts.

By Jin Mu, Guanhua Chen
arXiv Machine Learning
Sep 21

Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA

The paper presents the first large‑scale benchmark for uncertainty quantification (UQ) calibration in long‑form scientific question answering, evaluating four UQ methods on 685,000 responses from up to 20 large language models across seven datasets. It shows that instruction tuning leads to token‑level probability polarization, undermining token‑level uncertainty signals, while reasoning model families differ in how they handle this effect. Only semantic consistency—consistency of the final answer—provides well‑calibrated outputs, demonstrating that semantic calibration remains robust in multi‑step, dependency‑rich reasoning.

By Philip M\"uller, Nicholas Popovi\v{c}, Michael F\"arber, Peter Steinbach
arXiv Computation and Language
Oct 1

SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

arXiv:2607.05721v2 Announce Type: replace Abstract: Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refin...

By Yimeng Zhang, Yingying Zhuang, Ziyi Wang, Yuxuan Lu, Pei Chen, Aman Gupta, Zhe Su, Ming Tan, Zhilin Zhang, Qun Liu, Manikandarajan Ramanathan, Rajashekar Maragoud, Edward Vul, Jing Huang, Dakuo Wang
arXiv AI
Sep 2

Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

The paper introduces Iterative MBR Distillation for Error Span Detection (ESD) in machine translation, a self‑evolution framework that replaces human annotations with pseudo‑labels generated by a large language model. By iteratively applying Minimum Bayes Risk decoding, the method produces high‑quality error spans without costly human effort. Experiments on WMT Metrics Shared Task datasets show that models trained solely on these pseudo‑labels outperform both unadapted baselines and supervised models trained on human data at system and span levels, while keeping sentence‑level performance competitive.

By Boxuan Lyu, Haiyue Song, Zhi Qu