arXiv AI

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

CIBuzzBench is a new benchmark that tests cross‑lingual understanding of Chinese internet buzzwords by providing 3,001 buzzwords with English explanations, equivalents, category labels, and harmfulness annotations. The benchmark defines three tasks—Meaning Explanation, Cross‑lingual Equivalent Matching, and Culturally Grounded Harmfulness Detection—to evaluate how well large language models translate and interpret these culturally nuanced terms. Experiments show that current state‑of‑the‑art LLMs still struggle with fine‑grained non‑literal meanings, robust matching, and accurate harmfulness detection across languages.

arXiv Computation and Language

Sep 4

Benchmarking Machine Translation on Chinese Social Media Texts

The paper introduces CSM-MTBench, a benchmark for evaluating machine translation on Chinese social media text. It addresses two main challenges: limited parallel data due to slang and stylistic nuances, and inadequate evaluation metrics that miss these informal features. The benchmark includes two expert-curated subsets—Fun Posts and Social Snippets—and proposes specialized evaluation methods for each, revealing significant differences among over 20 MT models in handling semantic and stylistic aspects.

By Kaiyan Zhao, Zheyong Xie, Zhongtao Miao, Xinze Lyu, Yao Hu, Shaosheng Cao

llms rag nlp benchmarks

arXiv AI

Oct 12

Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon

The paper introduces C$^2$TU, a training‑free and prompt‑free method for uncovering Chinese cloaked toxic content. It uses substring matching with a Chinese homophone graph and toxic lexicon to spot candidate toxic words, then filters non‑toxic candidates and corrects cloaks using two variants—one based on BERT and another on LLMs that overcome auto‑regressive limitations. Experiments on two Chinese toxic datasets show C$^2$TU outperforms the best competitor by up to 71% in F1 score and 35% in accuracy.

By Xuchen Ma, Jianxiang Yu, Wenming Shao, Bo Pang, Xiang Li

llms safety

arXiv Computation and Language

Sep 15

Beyond Surface Forms: A Comprehensive, Mechanism-Oriented Taxonomy of Indirect Linguistic Encoding for LLM-Based Coded Language Detection

arXiv:2606.27314v2 Announce Type: replace Abstract: To avoid moderation and surveillance on social media, some users routinely invent indirect linguistic expressions (ILE) that camouflage sensitive m...

By Hamid Reza Firoozfar, Mohammadsadegh Abolhasani, Reza Mousavi, Paul Jen-Hwa Hu

llms benchmarks safety

arXiv AI

Aug 18

VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation

arXiv:2608. 15600v1 Announce Type: new Abstract: The widespread circulation of abusive online content has increased the need for reliable moderation of Chinese social-media text.

By Mingyu Yuan, Shengtao Wen, Lingbing Guo, Zhen Bi, Xiang Chen

llms benchmarks safety

arXiv AI

Jun 16

CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment

arXiv:2606. 15396v1 Announce Type: cross Abstract: Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.

By Wenbo Yu, Bohua Wang, Hao Fang, Kuofeng Gao, Jingru Zeng, Xiaochen Yang, Tianyi Zhang, Xiaoxiao Ma, Jiawei Kong, Hao Wu, Bin Chen, Shu-Tao Xia, Min Zhang

llms rag benchmarks safety

arXiv AI

Aug 5

ChiEngMixBench: Evaluating Large Language Models on Expert-Style Chinese-English Terminology Mixing

arXiv:2601. 16217v2 Announce Type: replace-cross Abstract: Large language models increasingly mediate multilingual professional communication, where useful generation requires adapting to community conventions about which expressions are retained, translated, or mixed.

By Qingyan Yang, Tongxi Wang, Yunsheng Luo

llms benchmarks

arXiv AI

Oct 2

Verbal tics in frontier language models: A critical review of current releases, research evidence, and public discussion

arXiv:2604.19139v4 Announce Type: replace-cross Abstract: Repeated praise, canned reassurance, familiar contrasts, and conspicuous vocabulary are recurring subjects in discussions of large language m...

By Shuai Wu, Xue Li, Zhijun Wang, Bolun Liu, Weilin Cai, Zihao Su, Ran Wang

llms benchmarks

arXiv Computation and Language

Oct 23

Harm Is Behavioural, Moderation Is Lexical: How Language and Culture Decide Whom Automated Safety Systems Protect and Whom They Silence

The paper examines how automated safety systems unevenly flag harmful content, disproportionately labeling African American English as offensive and misclassifying content from drag queens and white nationalists. It identifies seven failure modes rooted in vocabulary-based harm detection and proposes an evaluation protocol that highlights gaps in performance across languages. Applied to five open systems in eleven languages, the study finds that none achieve native-speaker accuracy, and higher capability often increases discrimination, widening the disparity between best- and worst-served language communities.

By Gabriel Lars Sabadin, Dr Nicola Harding, Sarah Kingston

safety

arXiv Computation and Language

Nov 4

CULTURE-CMT: Benchmarking Contextualized Social Media Translation

CULTURE-CMT is a Chinese‑English benchmark that evaluates contextualized translation of social‑media comments by providing the original post and conversational context. It contains 572 manually verified examples grouped into Slang, Context, and Both, allowing diagnostic analysis of translation difficulty. The benchmark also introduces a retrieval‑augmented cultural‑effectiveness Judger that combines a calibrated scoring model with dynamically retrieved cultural knowledge.

By Linjuan Wu, Dingming Li, Yiwen Qiu, Xinze Lyu, Ye Guo, Daoxin Zhang, Zhe Xu, Weiming Lu

rag benchmarks

arXiv Computation and Language

Sep 17

TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation

arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating l...

By Jianan Wu

llms benchmarks

arXiv AI

Oct 15

An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution

arXiv:2610.15795v1 Announce Type: cross Abstract: Automated moderation systems are deployed at scale, yet they remain opaque to the users and moderators affected by their decisions, and their coverag...

By Muhammad Shahid Iqbal Malik

llms fine-tuning benchmarks safety

Hugging Face Trending Papers

Sep 3

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

IndicSafeEval is a new evaluation framework that tests the safety robustness of large language models against persuasion-based jailbreak attacks in Indian languages. The benchmark covers ten safety-critical content categories, six persuasive strategies, and four languages—Hindi, Bengali, Marathi, and Punjabi—producing 7,200 adversarial prompts. Experiments show that model safety varies significantly across languages, prompt styles, and risk categories, highlighting gaps in current English-centric safety assessments.

llms benchmarks safety

Related stories