← Back to all news
arXiv AI By Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong, Siying Hu, Haichang Li, Ziying Wang, Zhicong Lu

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Read the original on arXiv AI →

The paper introduces a benchmark for testing large language models (LLMs) on their ability to infer social pragmatic meanings in indirect and playful Chinese online comments. Using over 200,000 public social media interactions, the authors created 4,735 human-validated diagnostic items that pair a target comment with its preceding context and plausible misreadings. Eight LLMs were evaluated in a cross-writer setting, with the best model achieving 81.42% leave-writer-out accuracy, while human accuracy reached 90.8%. The study finds that models can detect broad irony or playfulness but often misidentify the specific mechanism or interactional move.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 4

Benchmarking Machine Translation on Chinese Social Media Texts

The paper introduces CSM-MTBench, a benchmark for evaluating machine translation on Chinese social media text. It addresses two main challenges: limited parallel data due to slang and stylistic nuances, and inadequate evaluation metrics that miss these informal features. The benchmark includes two expert-curated subsets—Fun Posts and Social Snippets—and proposes specialized evaluation methods for each, revealing significant differences among over 20 MT models in handling semantic and stylistic aspects.

By Kaiyan Zhao, Zheyong Xie, Zhongtao Miao, Xinze Lyu, Yao Hu, Shaosheng Cao
llmsragnlpbenchmarks
More like this →
arXiv Computation and Language
Oct 16

APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models

APRiL is an open‑ended benchmark designed to evaluate Chinese large language models’ social awareness. It comprises 2,196 evaluation instances across six categories, split into two tasks: Social Scenario Analysis, which tests understanding of social cues, mental states, and conflict structures, and Socially Aware Response, which checks whether models can generate appropriate, natural replies. Experiments on twelve models from six families show a consistent gap between understanding and response quality, indicating that stronger scenario analyses tend to improve response performance.

By Tian Lan, Han Zhang, Shanshan Wang, Yahan Yu, Jiang Li, Yugo Murawaki
llmsbenchmarks
More like this →
arXiv AI
Sep 21

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

CIBuzzBench is a new benchmark that tests cross‑lingual understanding of Chinese internet buzzwords by providing 3,001 buzzwords with English explanations, equivalents, category labels, and harmfulness annotations. The benchmark defines three tasks—Meaning Explanation, Cross‑lingual Equivalent Matching, and Culturally Grounded Harmfulness Detection—to evaluate how well large language models translate and interpret these culturally nuanced terms. Experiments show that current state‑of‑the‑art LLMs still struggle with fine‑grained non‑literal meanings, robust matching, and accurate harmfulness detection across languages.

By Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu, Fanliang Bu
llmsbenchmarkssafety
More like this →