The paper analyzes how malicious actors create and share synthetic non-consensual intimate images (SNCII) across online forums, focusing on 4chan and Reddit. By examining over 280,000 4chan comments and 78,000 Reddit posts, the study identifies a wide array of resources—including 156 LoRA files targeting 141 female celebrities and 54 commercial nudification apps—used by both highly technical and less sophisticated users. The findings show that knowledge transfer between experts and newcomers fuels the spread of these illicit resources, revealing gaps in current regulatory frameworks and suggesting intervention points to deter such activity.
By Bernardo B. P. Medeiros (University of Florida), Malvika Jadhav (University of Florida), Allison Lu (University of Florida), Tadayoshi Kohno (Georgetown University), Vincent Bindschaedler (University of Florida), Kevin R. B. Butler (University of Florida)
The study audits Bluesky’s Moderation Service (BMS) using its 10.6 million public moderation labels from 2025. It finds that BMS operates as a human‑AI collaboration: sexual and graphic content is flagged automatically in seconds, while more nuanced or high‑stakes content requires human review that can take hours or days. The system shows high precision (0.837) but low recall (0.222), with annotators detecting 4.5 times more harmful content than the system, and clustering reveals harms ranging from hostility toward protected groups to the spread of explicit material.
By Pushpdeep Singh, Sayeh Jarollahi, Ayan Majumdar, Vabuk Pahari, Abhijnan Chakraborty, Krishna P. Gummadi, Ingmar Weber, Abhisek Dash
The study audits hate‑speech moderation on Twitter (now X) using 540,000 annotated tweets from a full day. Eighty percent of hateful tweets, including violent content, remained online after five months, and removal was only slightly more likely than for non‑hateful tweets, far below the rates for scams or adult content. Automated detection could not reliably classify hate but ranked it highly, allowing human triage; however, current staffing curbed little exposure, while substantial reductions were financially feasible and far below applicable regulatory fines.
By Manuel Tonneau, Dylan Thurgood, Diyi Liu, Niyati Malhotra, Victor Orozco-Olvera, Ralph Schroeder, Scott A. Hale, Manoel Horta Ribeiro, Paul R\"ottger, Samuel P. Fraiberger
The paper investigates how large language models (LLMs) used as labelers for toxic content differ from human crowd labels, focusing on the bias that LLMs, particularly Claude Haiku, over‑flag second‑person address (“you”) as toxic. On a dataset of 1.25 million news comments, Haiku incorrectly flags a much higher proportion of non‑toxic comments containing “you” compared to human raters, leading to a false increase in measured toxicity with reply depth. The authors show that rewriting “you” in the third person removes most of this bias, and that a certification approach—estimating laws rather than trusting individual labelers—can produce unbiased toxicity metrics even when using LLM scores.
"Why it matters": The study reveals that relying on LLM labelers without correcting for their bias can distort our understanding of toxic discourse online, potentially leading to misguided moderation policies.
By Liang Yue, Eric A. Vance
arXiv:2602. 02838v2 Announce Type: replace-cross Abstract: The detection of online influence operations -- coordinated campaigns by malicious actors to spread narratives -- has traditionally depended on content analysis or network features.
By Philipp J. Schneider, Lanqin Yuan, Marian-Andrei Rizoiu
arXiv:2607. 26236v1 Announce Type: cross Abstract: AI-generated counterspeech offers a scalable and effective strategy to mitigate online toxicity by promoting more constructive dialogue.
By Lorenzo Cima, Alessio Miaschi, Amaury Trujillo, Marco Avenuti, Felice Dell'Orletta, Stefano Cresci
The paper presents a new taxonomy for online harm on video platforms, expanding existing frameworks to multimodal contexts and defining six content-based categories: Information, Hate and harassment, Addictive, Clickbait, Sexual, and Physical harms. It also evaluates multimodal large language models (MLLMs) as alternative annotators, comparing GPT‑4‑Turbo, non‑trained crowdworkers, and trained human annotators on a dataset of 19,422 potentially harmful YouTube videos. The study finds that MLLMs can perform comparably to human annotators in both binary and multi‑label classification tasks, suggesting potential applications in content moderation and social science research.
By Claire Wonjeong Jo, Miki Weso{\l}owska, Magdalena Wojcieszak
The study introduces a unified method for measuring rhetorical distortion across six social media platforms and two languages, distinguishing between creator posts and their replies. It identifies five distortion dimensions—Significance Inflation, Anxiety Manufacturing, Novelty Claims, Loaded Language, and Temporal Distortion—using a three‑tier pipeline that combines lexical rules, negative filters, and LLM verification. Results show that distortion is far more strongly linked to the platform than content type, with X/Twitter exhibiting the highest distortion scores, and that the audience layer (replies) carries significantly more distortion than creator posts.
By Xucheng Yu, Jiangping Chen, Haohan Wang
The paper presents a red‑teaming framework to evaluate how open‑source large language models (LLMs) can be used for political influence, focusing on models that can be deployed locally without weight modifications. It measures LLM Overton Windows (OWs) to assess expression fidelity of sampled political opinions and examines how natural‑language jailbreaks alter these scores. The study evaluates over 30 LLMs from 10 families across five countries, finding that left‑leaning content is expressed with higher fidelity, larger models tend to have lower mean OW scores, and jailbreak effectiveness varies significantly between model families.
By Daniel C. Ruiz, Anna Serbina, Ashwin Rao, Emilio Ferrara, Luca Luceri
arXiv:2606. 05256v1 Announce Type: new Abstract: This study analyzes a publicly released dataset from a discontinued field experiment on Reddit's r/ChangeMyView.
By Kokil Jaidka, Saifuddin Ahmed
The paper introduces JudgeGPT and RogueGPT, a dual‑axis framework that separates authenticity from attribution to study how people detect hallucinations and disinformation from foundation models. Using 918 evaluations on five models, including GPT‑4 and Llama‑2, the authors apply Structural Causal Models to test causal hypotheses about detection accuracy. Results show political orientation has little effect, while familiarity with fake news mediates detection, and a "fluency trap" in GPT‑4 outputs makes them hard to distinguish from human text, implying pre‑bunking should focus on cognitive source monitoring.
By Alexander Loth, Martin Kappes, Marc-Oliver Pahl
The study evaluates large language models (LLMs) that generate political networks by comparing them to survey-reported partisan exposure from the ANES 2020–2022 Social Media Study. It finds that while network structure and political assortativity vary across model configurations, many models either over‑separate Democratic and Republican egos or underestimate Republican exposure, leading to group‑specific errors in local partisan environments. These discrepancies persist even among networks with similar aggregate properties, indicating that aggregate metrics alone can mask important individual‑level inaccuracies.
By Marco Tonin, Michele Tizzoni, Bruno Lepri, Luca Maria Aiello