NodeSyn is a method that generates socially aligned synthetic data for evaluating AI models. It combines a fine‑tuned taxonomy generator with credible sources and sensitive attributes to produce queries that reflect real societal contexts and nuanced harms. In tests across ten LLMs in Medical Advice and Self‑Harm, NodeSyn outperformed generic synthetic baselines, raising failure rates significantly and revealing that failures stem from both topic selection and sociotechnical grounding.
By Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang, Yanzhou Pan, Erin van Liemt, Darlene Neal, Kshitij Pancholi, Jamila Smith-Loud
The paper introduces ImpactBench, an open, expert-guided platform designed to measure the multi-dimensional impact of generative AI on users. It evaluates AI models across psychological, physical, and social wellbeing metrics derived from domain experts and existing benchmarks, using simulated multi-turn conversations with teen and adult scenarios. Ten leading LLMs were tested on 48,540 conversations, showing 68.7% promotion of desirable behaviors but only 53.1% avoidance of harms, with an 83.3% agreement rate between automated judges and expert reviewers.
By Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn
arXiv:2609.22696v1 Announce Type: new
Abstract: Decentralized social media platforms create new opportunities and challenges for computational mental health research because data access, moderation,...
By Gaurab Chhetri, Anandi Dutta, Subasish Das
arXiv:2606. 18203v1 Announce Type: cross Abstract: The LLM-empowered personal health agents with user health (sensor) metrics have offered a promising pathway to alleviate global disparities in healthcare access.
By Weizhi Zhang, Zechen Li, Hamid Palangi, Ben Graef, A. Ali Heydari, Simon A. Lee, Salman Rahman, Ray Luo, Zeinab Esmaeilpour, Erik Schenck, Chloe Zhang, Yamin Li, Menglian Zhou, Philip S. Yu, Daniel McDuff, Lindsey Sunden, Mark Malhotra, Shwetak Patel, Ahmed A. Metwally
arXiv:2607. 12336v1 Announce Type: cross Abstract: Artificial Intelligence (AI) technologies, while serving as a foundational enabler for modern social media and digital health services, exert a bivalent effect by simultaneously acting as a combatant against and a spread vector for misinformation.
By Farnaz Farid, Raihan Alam, Al Al-Areqi, Farhad Ahamed, Muhammad Hassan Khan, Sadia Hossain, Irena Veljanova, Anika Tabassum Binte Hossain
WildSEEK is a new dataset of 3,000 real user information‑seeking queries, manually annotated for risk‑sensitive domains and whether the query is factoid or analytical. The accompanying evaluation framework tests LLM responses against four failure criteria—sycophantic behavior, overreliance, a default US‑centric perspective, and poor handling of vulnerable populations—finding higher failure rates for analytical queries. The authors also train classifiers on WildSEEK to analyze over 1.8 million realistic queries, revealing that more than a third are high‑risk and often analytical.
By Tanise Ceron, Joachim Baumann, Elisa Bassignana, Berat Cabuk, Dirk Hovy, Debora Nozza