The authors present the Cross-Platform Fairness Evaluation (CPFE) framework, a five‑axis audit protocol that assesses discriminative performance, calibration, statistical significance, prediction equity, and attribution stability of transformer models. Applying CPFE to four models trained on a Kaggle mental‑health corpus and tested on Reddit and Twitter, they find substantial cross‑platform degradation in AUC (30–40%) and severe calibration failures (ECE rising to 0.5 on Twitter). The study demonstrates that platform‑specific temperature scaling can largely fix calibration without harming discrimination, while prediction equity and attribution stability analyses reveal significant disparities and vocabulary divergence across platforms. The results argue that cross‑platform validation across all CPFE axes should become a standard requirement for mental‑health NLP systems deployed in heterogeneous environments.
By Rajveer Singh Pall, Sameer Yadav
arXiv:2605.30273v2 Announce Type: replace-cross
Abstract: Large language models (LLMs) show promise in generating supportive responses for mental health queries, but improving their usefulness, empat...
By Jiwon Kim, Maya Ajit, Sherry Gong, Soorya Ram Shimgekar, Dong Whi Yoo, Eshwar Chandrasekharan, Koustuv Saha
arXiv:2512. 06227v3 Announce Type: replace-cross Abstract: Real-world indicators play an important role in many Natural Language Processing (NLP) applications, such as life events for mental health analysis and risky behaviours for online safety, yet labelling such information is often costly and/or difficult due to its multi-label and dynamic nature.
By Junyu Mao, Anthony Hills, Talia Tseriotou, Maria Liakata, Aya Shamir, Dan Sayda, Dana Atzil-Slonim, Natalie Djohari, Pamela Ugwudike, Mahesan Niranjan, Stuart E. Middleton
NodeSyn is a method that generates socially aligned synthetic data for evaluating AI models. It combines a fine‑tuned taxonomy generator with credible sources and sensitive attributes to produce queries that reflect real societal contexts and nuanced harms. In tests across ten LLMs in Medical Advice and Self‑Harm, NodeSyn outperformed generic synthetic baselines, raising failure rates significantly and revealing that failures stem from both topic selection and sociotechnical grounding.
By Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang, Yanzhou Pan, Erin van Liemt, Darlene Neal, Kshitij Pancholi, Jamila Smith-Loud
NodeSynth is a method that generates socially aligned synthetic data by combining a fine‑tuned taxonomy generator with credible sources and extracted sensitive attributes to produce queries that reflect real societal contexts and nuanced harms. In evaluations across ten open‑weight and proprietary large language models on Medical Advice and Self‑Harm tasks, NodeSynth outperformed generic synthetic baselines, structured synthetic benchmarks, and authentic human‑authored queries, inducing higher failure rates that highlight model vulnerabilities. Ablation and variance attribution studies show that the granular Level 3 taxonomic expansion and sociotechnical grounding of queries drive these failure rates, underscoring the importance of nuanced data for high‑stakes evaluation.
By Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang, Yanzhou Pan, Erin van Liemt, Darlene Neal, Kshitij Pancholi, Jamila Smith-Loud
The paper presents a system for the IEEE BigData 2026 Cup on Explainable Suicide Risk Assessment on Social Media. It tackles three tasks—risk-level classification, evidence phrase extraction, and multi-label factor identification—using Qwen2.5-Instruct models adapted with quantized low-rank adaptation (QLoRA) and an answer-masked causal language-model objective. The final system achieved a composite score of 0.7738, with 0.8089 on Task 1 and 0.6919 on Task 2, demonstrating that task‑specific training and tailored aggregation improve performance across the three tasks.
By Xuan Zhong Feng, Geoffrey Martin, Hexin Dong, Yifan Peng
arXiv:2508. 00923v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to answer health-related questions and support healthcare workflows, yet evidence for their safety still relies heavily on static benchmarks that can rapidly become obsolete or be optimized against.
By Jiazhen Pan (Cherise), Bailiang Jian (Cherise), Paul Hager (Cherise), Yundi Zhang (Cherise), Che Liu (Cherise), Friederike Jungmann (Cherise), Hongwei Bran Li (Cherise), Julian Canisius (Cherise), Chenyu You (Cherise), Junde Wu (Cherise), Jiayuan Zhu (Cherise), Fenglin Liu (Cherise), Yuyuan Liu (Cherise), Niklas Bubeck (Cherise), Moritz Knolle (Cherise), Chen (Cherise), Chen (Cherise), Christian Wachinger, Zhenyu Gong, Cheng Ouyang, Georgios Kaissis, Benedikt Wiestler, Daniel Rueckert
The study audits Bluesky’s Moderation Service (BMS) using its 10.6 million public moderation labels from 2025. It finds that BMS operates as a human‑AI collaboration: sexual and graphic content is flagged automatically in seconds, while more nuanced or high‑stakes content requires human review that can take hours or days. The system shows high precision (0.837) but low recall (0.222), with annotators detecting 4.5 times more harmful content than the system, and clustering reveals harms ranging from hostility toward protected groups to the spread of explicit material.
By Pushpdeep Singh, Sayeh Jarollahi, Ayan Majumdar, Vabuk Pahari, Abhijnan Chakraborty, Krishna P. Gummadi, Ingmar Weber, Abhisek Dash
The paper reviews how large language models are applied in mental health, covering areas such as social media analysis, clinical conversational agents, therapy support tools, prompt engineering, and multimodal learning. It synthesizes interdisciplinary studies that use social media posts, electronic medical records, and multimodal inputs to detect depression, assess suicide risk, provide personalized therapy, and generate psychoeducational content. The review also discusses advances in model interpretability, annotation strategies, multimodal fusion techniques, and highlights ethical, sociotechnical, and regulatory challenges while proposing frameworks for safe, equitable, and accountable deployment.
By Yisong Chen, Yifan Gao, Sijing Yu, Chuqing Zhao, Yang Lu
The paper presents a new benchmark for automatically evaluating mental health stigma in online text, featuring a fine‑grained taxonomy that covers stigma mode, domain, and specific components across multiple mental health conditions. The authors annotate naturally occurring news and social media posts and test large language models and classifiers for sentiment, toxicity, and hate speech, finding that these models poorly capture stigma and often overpredict it without explicit rules. The benchmark, annotations, exemplar cases, and code are publicly released on GitHub.
By Naomi Baes, Jemima Kang, Nick Haslam, Chris Groot, Alsa Wu, Luc Raszewski, Yulia Otmakhova
arXiv:2609.07766v1 Announce Type: cross
Abstract: Assessing suicide risk from social media text is a small-data, high-stakes setting requiring not only severity prediction but also supporting evidenc...
By Shlok Shelat, Shrey Salvi, Souvik Roy, Manas Gaur, Amit Sheth
arXiv:2606. 28334v1 Announce Type: cross Abstract: Recent advances in artificial intelligence (AI) and social media data have led to growing optimism about the ability to detect suicide risk at scale.
By Yaakov Ophir, Ofri Hefetz, Refael Tikochinski, Kfir Bar, Shir Lissak, Shulamit Grinapol, Haya Wachtel, Eyal Fruchter, Roi Reichart