arXiv AI

The Divergence Hypothesis: Unmasking Lexical Interference and Label Bias in Mental Health NLP

The paper introduces TSS (Triple-Stream Stress probe), a diagnostic framework that splits text into lexical, morpho-syntactic, and psycholinguistic style channels to analyze mental health NLP classifiers. Across four English datasets, TSS uncovers a lexical interference effect where adding lexical features harms performance on human-labeled data but not on auto-labeled data, and proposes the Degree of Divergence (DoD) statistic to audit label-source bias. The study demonstrates that style features largely remain effective even after masking content words, emphasizing that shortcut learning is label-source specific rather than clinically relevant.

arXiv Machine Learning

2d ago

Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media

The authors present the Cross-Platform Fairness Evaluation (CPFE) framework, a five‑axis audit protocol that assesses discriminative performance, calibration, statistical significance, prediction equity, and attribution stability of transformer models. Applying CPFE to four models trained on a Kaggle mental‑health corpus and tested on Reddit and Twitter, they find substantial cross‑platform degradation in AUC (30–40%) and severe calibration failures (ECE rising to 0.5 on Twitter). The study demonstrates that platform‑specific temperature scaling can largely fix calibration without harming discrimination, while prediction equity and attribution stability analyses reveal significant disparities and vocabulary divergence across platforms. The results argue that cross‑platform validation across all CPFE axes should become a standard requirement for mental‑health NLP systems deployed in heterogeneous environments.

By Rajveer Singh Pall, Sameer Yadav

llms nlp fine-tuning safety

arXiv Machine Learning

Aug 4

Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models

arXiv:2608. 00144v1 Announce Type: new Abstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone.

By Victor Maricato

llms safety

arXiv AI

Jul 7

The Rise of Verbal Tics in Large Language Models: A Systematic Analysis Across Frontier Models

arXiv:2604. 19139v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) continue to evolve through alignment techniques such as Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI, a growing and increasingly conspicuous phenomenon has emerged: the proliferation of verbal tics--repetitive, formulaic linguistic patterns that pervade model outputs.

By Shuai Wu, Xue Li, Yanna Feng, Yufang Li, Zhijun Wang, Ran Wang

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 7

Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)

arXiv:2607. 04223v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why.

By Mohamed Aly Bouke

rag nlp benchmarks safety

arXiv AI

Jul 28

LEX-EC: A Lexical Evidence-Channel Audit Framework for Zero-Shot LLM Personality Classification in Black-Box Settings

arXiv:2607. 24435v1 Announce Type: cross Abstract: Large language models may easily assign personality labels from text, but model interpretability remains an open problem.

By Brittany Harbison, Ashok K. Goel

llms benchmarks safety

arXiv Machine Learning

Jun 9

Priors Persist Through Suppression: A Stroop Paradigm for Lexical Override

arXiv:2606. 07555v1 Announce Type: cross Abstract: Glossaries, technical specifications, and system prompts routinely ask language models to use familiar words in unfamiliar ways.

By Han-yu Wang

llms nlp

arXiv Machine Learning

Jul 7

The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices

arXiv:2603. 18482v2 Announce Type: replace-cross Abstract: Standard decoding strategies for text generation, including top-$k$, nucleus sampling, and contrastive search, select tokens based on likelihood, restricting outputs to high-probability regions.

By Esteban Garces Arias, Nurzhan Sapargali, Christian Heumann, Matthias A{\ss}enmacher

llms safety

arXiv Machine Learning

Jun 5

Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries

arXiv:2606. 05970v1 Announce Type: cross Abstract: Large language models are increasingly used for structured extraction from clinical free-text notes, but the sensitivity of their output to upstream configuration choices is less understood than their accuracy on fixed benchmarks.

By Martin Murin

llms benchmarks

arXiv Machine Learning

Aug 12

Leak It: Per-Document Extraction Beyond Aggregate Membership Inference

arXiv:2608. 00144v2 Announce Type: replace Abstract: Membership inference (MIA) on language models is usually summarised by aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines can separate members from non-members using surface text alone.

By Victor Maricato

llms safety

arXiv AI

Jun 24

SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization

arXiv:2606. 24259v1 Announce Type: cross Abstract: Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge.

By Noor Islam S. Mohammad, Ulug Bayazit

llms nlp fine-tuning efficiency safety

arXiv AI

Jul 2

Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation

arXiv:2607. 01208v1 Announce Type: cross Abstract: Language models deployed in high-stakes roles can potentially favor certain entities, brands, or viewpoints, steering user decisions at scale.

By Shayan Talaei, Abhinav Chinta, Devvrit Khatri, Amin Karbasi, Azalia Mirhoseini, Amin Saberi

llms efficiency safety

arXiv Machine Learning

Jul 21

Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation

arXiv:2604. 16197v2 Announce Type: replace Abstract: Data attribution and valuation are critical for understanding data-model synergy for Large Language Models (LLMs), yet existing gradient-based methods suffer from scalability challenges on LLMs.

By Yide Ran, Jianwen Xie, Minghui Wang, Wenjin Zheng, Denghui Zhang, Chuan Li, Zhaozhuo Xu

llms

Related stories