arXiv AI September 15, 2026 By Darin Keng, Zhewei Sun

Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

The paper investigates how large language models handle domain-specific jargon, comparing a general-purpose Llama‑3.1 with a version fine‑tuned on medical data. Two new medical jargon benchmarks reveal that the general model actually outperforms the fine‑tuned variant, and interpretability tools show the fine‑tuned model over‑emphasizes a few components linked to jargon predictions. Reweighting these components narrows the performance gap, and some jargon‑sensitive components also aid materials‑science tasks, indicating a partially domain‑agnostic representation of specialized terminology.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 22

MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications

arXiv:2409. 07314v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows.

By Praveenkumar Kanithi, Cl\'ement Christophe, Marco AF Pimentel, Tathagata Raha, Prateek Munjal, Nada Saadi, Hamza A Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan

llms fine-tuning benchmarks safety data-engineering

arXiv AI

Jul 21

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.

By Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, C\'eline Hudelot, Pierre Colombo

llms

arXiv AI

Jun 8

When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations

arXiv:2606. 07237v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in healthcare for tasks such as clinical question answering, diagnosis support, and report summarization.

By Mahdi Alkaeed

llms nlp benchmarks safety

arXiv AI

Sep 1

SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence

arXiv:2608.29582v1 Announce Type: cross Abstract: Current evaluations of large language models (LLMs) primarily focus on factual knowledge retrieval, overlooking the fundamental challenge of navigati...

By Yi Yu, Bo Wang, Chong Feng, Ge Shi, Xia Liu, Ziyi Yang, Xuewen Shi

llms benchmarks safety

arXiv AI

Jun 2

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

arXiv:2606. 00467v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions.

By Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez

llms rag safety

arXiv AI

Aug 19

Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation

The paper investigates Retrieval-Augmented Generation fine‑tuning (RAG‑SFT) for generating requirements documents in electronics engineering, comparing two 7B models trained with different data strategies. It introduces a claim‑based evaluation pipeline, C‑FEX, and a new metric, Parametric Knowledge Precision (PKP), to assess factuality of model‑generated claims. Results show that fine‑tuned 7B models can match or surpass a 72B baseline, but standard metrics may mislead, and fine‑tuning reduces hallucination by encouraging more reliable use of parametric knowledge.

By Julian Oestreich, Maximilian Bley, Frank Binder, Lydia M\"uller, Andr\'e Alcalde, Maksym Sydorenkoq

rag nlp fine-tuning safety

One email a morning, machine-written

Related stories

MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations

SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation