← Back to all news
arXiv Computation and Language By Ashanvi Yadav, Shubham Bhardwaj

Used, Mentioned, or Condemned? A Controlled Contrast-Set Diagnostic for the Use-Mention Distinction in Code-Mixed Hinglish Misogyny Detection

Read the original on arXiv Computation and Language →

The paper introduces a diagnostic tool for distinguishing the use of misogynistic slurs from their mention in counter‑speech within code‑mixed Hinglish. It identifies evaluation artifacts in existing corpora, releases a 416‑item minimal‑pair contrast set that decorrelates slur presence and gendered register from labels, and proposes a pair‑consistency metric to assess model performance. Experiments show that even strong baselines struggle to consistently label counter‑speech pairs, while a large language model achieves perfect scores, indicating the benchmark measures genuine capability rather than exploitation of artifacts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Oct 23

Harm Is Behavioural, Moderation Is Lexical: How Language and Culture Decide Whom Automated Safety Systems Protect and Whom They Silence

The paper examines how automated safety systems unevenly flag harmful content, disproportionately labeling African American English as offensive and misclassifying content from drag queens and white nationalists. It identifies seven failure modes rooted in vocabulary-based harm detection and proposes an evaluation protocol that highlights gaps in performance across languages. Applied to five open systems in eleven languages, the study finds that none achieve native-speaker accuracy, and higher capability often increases discrimination, widening the disparity between best- and worst-served language communities.

By Gabriel Lars Sabadin, Dr Nicola Harding, Sarah Kingston
safety
More like this →
arXiv Computation and Language
Sep 15

Beyond Surface Forms: A Comprehensive, Mechanism-Oriented Taxonomy of Indirect Linguistic Encoding for LLM-Based Coded Language Detection

arXiv:2606.27314v2 Announce Type: replace Abstract: To avoid moderation and surveillance on social media, some users routinely invent indirect linguistic expressions (ILE) that camouflage sensitive m...

By Hamid Reza Firoozfar, Mohammadsadegh Abolhasani, Reza Mousavi, Paul Jen-Hwa Hu
llmsbenchmarkssafety
More like this →
arXiv Computation and Language
Nov 4

When the Text Is Silent: Memory Shortcuts and the Validity of LLM Annotation under Evidence Insufficiency

The study investigates whether large language models (LLMs) rely on textual evidence or on their pre‑existing knowledge when annotating text. Using counterfactual prompts for 995 public figures, the authors find that LLMs often default to memory shortcuts—providing correct facts about a figure even when the text contains no supporting evidence. These shortcuts are amplified by evaluation design and incentives, but can be mitigated by name reminders or masking, though with trade‑offs in gender annotation.

By Wenchang Wang, Yue Song
llms
More like this →
arXiv Computation and Language
Sep 1

When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speech

The paper introduces WSF-ARG+, a new dataset that pairs hate speech with check‑worthiness annotations, and presents an LLM‑in‑the‑loop framework to streamline the annotation process. Experiments with 12 open‑weight large language models demonstrate that the framework cuts human effort while maintaining annotation quality. The study also shows that incorporating check‑worthiness labels improves hate‑speech detection performance, boosting macro‑F1 scores for large models by up to 0.213 and averaging 0.154 across models.

By Nicol\'as Benjam\'in Ocampo, Tommaso Caselli, Davide Ceolin
llms
More like this →