arXiv Computation and Language
Handle with CARE: Can LLMs Reproduce How Online Communities React?
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 1
Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts
The study investigates how large language models (LLMs) assess psychological distress in online posts from six identity‑based communities. Through a perspectivist annotation task, 321 participants provided 9,587 judgments on 1,198 Reddit posts, revealing modest in‑group agreement (OR = 1.18) that varies across communities. When evaluated against these community‑specific labels, open‑weight LLMs consistently over‑estimate distress—achieving only 31–44% accuracy on posts perceived as none‑to‑mild—while newer models like GPT‑5 and Gemini 2.5 Pro show similar inflation, whereas Claude Opus 4 is more conservative. "whyItMatters":"The findings highlight that miscalibrated distress detection by LLMs can disproportionately impact the very communities they aim to serve, underscoring the need for equitable AI deployment in mental‑health contexts."
arXiv AI
Jun 16
MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions
arXiv:2606. 14715v1 Announce Type: cross Abstract: LLM agents are increasingly used to simulate real world interactions, but it remains unclear whether simulated behaviors preserve the content patterns and interaction dynamics of real human behaviors.
arXiv Computation and Language
Sep 1
Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions
arXiv:2606.06443v3 Announce Type: replace Abstract: Large language models are increasingly used to simulate social media users and infer how individuals may respond to online discussions. However, it...
arXiv Machine Learning
Jun 5
RedditPersona: A Modular Framework for Community-Conditioned LLM Adaptation from Reddit
arXiv:2606. 06027v1 Announce Type: cross Abstract: Community-conditioned language model adaptation requires choices about data collection, community definition, and evaluation that are currently made independently in each study, making it hard to compare assumptions or reuse artifacts.
arXiv AI
4d ago
ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions
ThreadBench is a benchmark that evaluates how well large language model (LLM) agents simulate real-world Reddit discussions, using 9,817 real threads across 8 domains and 116 subdomains. It compares generated and real threads on 12 metrics grouped into uniformity, expression, tone, and interaction. The authors also introduce Thread-Prompt-based Discussion Generation (ThreadPDG), a Planner-Writer framework that learns thread growth from real seeds and achieves high pass rates on ThreadBench, outperforming baseline methods and matching real discussions in 78.7% of human evaluations.
arXiv AI
Sep 21
Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation
The paper introduces the COmmunity-centered Peer Engaged Support (COPES) dataset and a three‑axis evaluation framework to gauge how well Large Language Models (LLMs) align with community perspectives on mental‑health support queries. Experiments show that fine‑tuning LLMs on COPES improves strategy alignment and emotion‑tone alignment by over 50% for general‑purpose models, yet these gains are uneven across subreddits and coping strategies. The study also finds that post‑training shifts the model’s recommendations toward problem‑focused advice while reducing emotion‑focused responses, indicating persistent disparities in performance across different communities and needs.