arXiv AI
Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 28
Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media
The authors present the Cross-Platform Fairness Evaluation (CPFE) framework, a five‑axis audit protocol that assesses discriminative performance, calibration, statistical significance, prediction equity, and attribution stability of transformer models. Applying CPFE to four models trained on a Kaggle mental‑health corpus and tested on Reddit and Twitter, they find substantial cross‑platform degradation in AUC (30–40%) and severe calibration failures (ECE rising to 0.5 on Twitter). The study demonstrates that platform‑specific temperature scaling can largely fix calibration without harming discrimination, while prediction equity and attribution stability analyses reveal significant disparities and vocabulary divergence across platforms. The results argue that cross‑platform validation across all CPFE axes should become a standard requirement for mental‑health NLP systems deployed in heterogeneous environments.
arXiv AI
Sep 10
LLUMI: Improving LLM Writing Assistance for Mental Health Support with Online Community Feedback
arXiv:2605.30273v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise in generating supportive responses for mental health queries, but improving their usefulness, empat...
arXiv Machine Learning
Aug 12
Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety
arXiv:2512. 06227v3 Announce Type: replace-cross Abstract: Real-world indicators play an important role in many Natural Language Processing (NLP) applications, such as life events for mental health analysis and risky behaviours for online safety, yet labelling such information is often costly and/or difficult due to its multi-label and dynamic nature.
arXiv Computation and Language
Oct 26
NodeSyn: Socially Aligned Synthetic Data for AI Evaluation
NodeSyn is a method that generates socially aligned synthetic data for evaluating AI models. It combines a fine‑tuned taxonomy generator with credible sources and sensitive attributes to produce queries that reflect real societal contexts and nuanced harms. In tests across ten LLMs in Medical Advice and Self‑Harm, NodeSyn outperformed generic synthetic baselines, raising failure rates significantly and revealing that failures stem from both topic selection and sociotechnical grounding.
arXiv Machine Learning
Oct 22
NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
NodeSynth is a method that generates socially aligned synthetic data by combining a fine‑tuned taxonomy generator with credible sources and extracted sensitive attributes to produce queries that reflect real societal contexts and nuanced harms. In evaluations across ten open‑weight and proprietary large language models on Medical Advice and Self‑Harm tasks, NodeSynth outperformed generic synthetic baselines, structured synthetic benchmarks, and authentic human‑authored queries, inducing higher failure rates that highlight model vulnerabilities. Ablation and variance attribution studies show that the granular Level 3 taxonomic expansion and sociotechnical grounding of queries drive these failure rates, underscoring the importance of nuanced data for high‑stakes evaluation.
arXiv Machine Learning
Oct 3
Explainable Suicide Risk Assessment on Social Media with Multi-Task QLoRA
The paper presents a system for the IEEE BigData 2026 Cup on Explainable Suicide Risk Assessment on Social Media. It tackles three tasks—risk-level classification, evidence phrase extraction, and multi-label factor identification—using Qwen2.5-Instruct models adapted with quantized low-rank adaptation (QLoRA) and an answer-masked causal language-model objective. The final system achieved a composite score of 0.7738, with 0.8089 on Task 1 and 0.6919 on Task 2, demonstrating that task‑specific training and tailored aggregation improve performance across the three tasks.