The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper introduces a benchmark for testing large language models (LLMs) on their ability to infer social pragmatic meanings in indirect and playful Chinese online comments. Using over 200,000 public social media interactions, the authors created 4,735 human-validated diagnostic items that pair a target comment with its preceding context and plausible misreadings. Eight LLMs were evaluated in a cross-writer setting, with the best model achieving 81.42% leave-writer-out accuracy, while human accuracy reached 90.8%. The study finds that models can detect broad irony or playfulness but often misidentify the specific mechanism or interactional move.
The paper introduces the Core Sentiment Inventory (CSI), a new personality trait evaluation tool for large language models (LLMs) that addresses reliability and validity issues found in existing methods like the Big Five Inventory (BFI). CSI is designed specifically for LLMs, supports both English and Chinese, and provides detailed psychological portraits of model behavior. Experiments show that CSI captures nuanced behavioral patterns, improves reliability, and correlates strongly (above 0.85) with real-world LLM outputs.
arXiv:2609.36194v1 Announce Type: new Abstract: Extracted sentiment directions can vary across samples even when downstream sentiment classification remains accurate. To evaluate direction reproducib...
The study evaluates how well three sentiment‑analysis tools (TextBlob, VADER, Twitter‑roBERTa‑base) and three large language models (Qwen3‑32B, GPT‑OSS‑120B, Llama‑4‑Maverick‑17B) agree with six human raters on 100 tweets. Agreement was measured with Cohen’s and Fleiss’ kappa, revealing only fair inter‑human agreement and higher concordance for binary sentiment labels than for three‑class labels. Twitter‑roBERTa‑base achieved the strongest alignment with humans, especially for negative versus non‑negative sentiment, while the LLMs showed substantial agreement among themselves and moderate to substantial alignment with humans, particularly for positive versus non‑positive classifications. The findings emphasize that domain‑specific fine‑tuning and human‑centered evaluation are essential for reliable social media sentiment analysis.
The paper "HateSense: Hierarchical Sentiment Analysis Framework for Hate Speech Detection: Implementing Binary and Multiclass Classification Strategy" revisits a transformer-based framework that combines hierarchical classification and sentiment features for hate speech detection. Experiments on the Davidson and HateXplain datasets show that a dominant-class-first hierarchy actually lowers macro‑F1, sentiment features have minimal impact, and class‑weighted training yields the largest performance gains. Cross‑dataset transfer and slur reuse reveal that in‑domain metrics overstate real‑world reliability.
The study investigates how different rating scale designs affect scores produced by large language models (LLMs) in text scoring tasks. By varying format, range, and sign across tasks like political leaning, sentiment, and toxicity, the authors find that a single design change can shift normalized scores by about 0.10 and alter the category of roughly 20% of texts. The research highlights that rubric-based prompting does not mitigate this sensitivity and emphasizes the need for robustness checks when using LLMs for scalar scoring.