The paper examines how automated safety systems unevenly flag harmful content, disproportionately labeling African American English as offensive and misclassifying content from drag queens and white nationalists. It identifies seven failure modes rooted in vocabulary-based harm detection and proposes an evaluation protocol that highlights gaps in performance across languages. Applied to five open systems in eleven languages, the study finds that none achieve native-speaker accuracy, and higher capability often increases discrimination, widening the disparity between best- and worst-served language communities.
By Gabriel Lars Sabadin, Dr Nicola Harding, Sarah Kingston
arXiv:2606.27314v2 Announce Type: replace
Abstract: To avoid moderation and surveillance on social media, some users routinely invent indirect linguistic expressions (ILE) that camouflage sensitive m...
By Hamid Reza Firoozfar, Mohammadsadegh Abolhasani, Reza Mousavi, Paul Jen-Hwa Hu
arXiv:2611.02491v1 Announce Type: new
Abstract: Large language models are increasingly used as measurement instruments for socially consequential constructs such as hate speech, and are typically val...
By Wenchang Wang, Yue Song
The study investigates whether large language models (LLMs) rely on textual evidence or on their pre‑existing knowledge when annotating text. Using counterfactual prompts for 995 public figures, the authors find that LLMs often default to memory shortcuts—providing correct facts about a figure even when the text contains no supporting evidence. These shortcuts are amplified by evaluation design and incentives, but can be mitigated by name reminders or masking, though with trade‑offs in gender annotation.
By Wenchang Wang, Yue Song
The paper introduces WSF-ARG+, a new dataset that pairs hate speech with check‑worthiness annotations, and presents an LLM‑in‑the‑loop framework to streamline the annotation process. Experiments with 12 open‑weight large language models demonstrate that the framework cuts human effort while maintaining annotation quality. The study also shows that incorporating check‑worthiness labels improves hate‑speech detection performance, boosting macro‑F1 scores for large models by up to 0.213 and averaging 0.154 across models.
By Nicol\'as Benjam\'in Ocampo, Tommaso Caselli, Davide Ceolin
arXiv:2610.15908v1 Announce Type: new
Abstract: Phase-annotated conversational data is essential for studying the progression of online grooming of minors and for developing early warning tools. Howe...
By Emanuele Florindi, Valentina Franzoni, Ludovico Guercio, Alfredo Milani, Mattia Polticchia
arXiv:2607. 26339v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) systems ground large language models (LLMs) in external corpora, but this reliance exposes them to corpus poisoning: maliciously injected passages that manipulate retrieved evidence.
By Pushkal Kumar, Tucker Nielson, Tanish Kolhe, Shubham Zala, Vincent Li
arXiv:2608. 13695v1 Announce Type: cross Abstract: Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe for non-English-speaking users.
By Chialuka Prisca-Mary Onuoha, Bright Etornam Sunu, Rashidat Sikiru
The paper introduces TSS (Triple-Stream Stress probe), a diagnostic framework that splits text into lexical, morpho-syntactic, and psycholinguistic style channels to analyze mental health NLP classifiers. Across four English datasets, TSS uncovers a lexical interference effect where adding lexical features harms performance on human-labeled data but not on auto-labeled data, and proposes the Degree of Divergence (DoD) statistic to audit label-source bias. The study demonstrates that style features largely remain effective even after masking content words, emphasizing that shortcut learning is label-source specific rather than clinically relevant.
By Moustafa Yehia Hassan
Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart.
arXiv:2608. 16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation.
By Yifan Zhang, Rahmatollah Beheshti
The paper presents an instruction‑tuned large language model (LLM) based on Qwen3 that is fine‑tuned for hate speech mitigation by unifying 36 English hate speech datasets. The authors show that this generalist LLM achieves state‑of‑the‑art performance on in‑domain benchmarks and delivers significant gains in cross‑domain and cross‑lingual generalization, outperforming specialist encoder‑based classifiers.
By Lukas Edman, Daryna Dementieva, Alexander Fraser