← Back to all news
arXiv Machine Learning By Rajveer Singh Pall, Sameer Yadav

Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media

Read the original on arXiv Machine Learning →

The authors present the Cross-Platform Fairness Evaluation (CPFE) framework, a five‑axis audit protocol that assesses discriminative performance, calibration, statistical significance, prediction equity, and attribution stability of transformer models. Applying CPFE to four models trained on a Kaggle mental‑health corpus and tested on Reddit and Twitter, they find substantial cross‑platform degradation in AUC (30–40%) and severe calibration failures (ECE rising to 0.5 on Twitter). The study demonstrates that platform‑specific temperature scaling can largely fix calibration without harming discrimination, while prediction equity and attribution stability analyses reveal significant disparities and vocabulary divergence across platforms. The results argue that cross‑platform validation across all CPFE axes should become a standard requirement for mental‑health NLP systems deployed in heterogeneous environments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 24

The Divergence Hypothesis: Unmasking Lexical Interference and Label Bias in Mental Health NLP

The paper introduces TSS (Triple-Stream Stress probe), a diagnostic framework that splits text into lexical, morpho-syntactic, and psycholinguistic style channels to analyze mental health NLP classifiers. Across four English datasets, TSS uncovers a lexical interference effect where adding lexical features harms performance on human-labeled data but not on auto-labeled data, and proposes the Degree of Divergence (DoD) statistic to audit label-source bias. The study demonstrates that style features largely remain effective even after masking content words, emphasizing that shortcut learning is label-source specific rather than clinically relevant.

By Moustafa Yehia Hassan
nlpsafety
More like this →
arXiv Computation and Language
Sep 21

Reading Anxiety or Reading the Label? Comparing Fine-Tuned and Frontier Models for Anxiety Detection on Social Media

The study compares six approaches—frontier commercial models, fine‑tuned smaller models, and conventional classifiers—for detecting anxiety in Reddit posts. It reveals a significant lexical bias: 69.3% of anxiety‑labelled posts contain the word "anxiety" or a variant, allowing models to perform well via keyword matching rather than true language understanding. After removing these terms, the frontier model still leads (F1 = 0.846), but a 110 M‑parameter domain‑adapted encoder achieves a close score (F1 = 0.831) without external API calls, and lexical dependence varies widely across models.

By Cris Huynh, Arlene Pham
fine-tuning
More like this →
arXiv Machine Learning
Sep 2

A Multi-Branch Feature Fusion Approach for Health Misinformation Detection and Propagation

This paper introduces a multi‑branch fusion framework that combines transformer‑based semantics, rhetorical cues, stance representations, and psychologically motivated proxies to detect health misinformation and characterize its spread on online social networks. The authors propose an interpretable Cognitive Propagation Score (CPS) derived from text cues that estimate argument complexity, emotional intensity, and virality potential, aiding diffusion‑risk reasoning when engagement data are missing. Experiments on three benchmark datasets (Constraint, COVID‑19_FNIR, Monkeypox) demonstrate near‑perfect classification and ranking performance, with ablation studies showing complementary gains from psychological and rhetorical components.

By Mkululi Sikosana, Sean Maudsley-Barton, Oluwaseun Ajao
llmsragdiffusionbenchmarkssafety
More like this →
arXiv AI
Aug 10

Natural Language Processing Psychometrics

arXiv:2608. 07316v1 Announce Type: cross Abstract: Natural Language Processing (NLP) models predicting mental health outcomes rarely specify what they measure: contextual knowledge, emotional content, or syntactic structure.

By Edoardo Sebastiano De Duro, Emma Franchino, Massimo Stella
llmsnlp
More like this →