The study evaluates how well three sentiment‑analysis tools (TextBlob, VADER, Twitter‑roBERTa‑base) and three large language models (Qwen3‑32B, GPT‑OSS‑120B, Llama‑4‑Maverick‑17B) agree with six human raters on 100 tweets. Agreement was measured with Cohen’s and Fleiss’ kappa, revealing only fair inter‑human agreement and higher concordance for binary sentiment labels than for three‑class labels. Twitter‑roBERTa‑base achieved the strongest alignment with humans, especially for negative versus non‑negative sentiment, while the LLMs showed substantial agreement among themselves and moderate to substantial alignment with humans, particularly for positive versus non‑positive classifications. The findings emphasize that domain‑specific fine‑tuning and human‑centered evaluation are essential for reliable social media sentiment analysis.
arXiv:2608. 15338v1 Announce Type: cross Abstract: Sentiment classifiers are increasingly applied to social media content that is either sarcastic or AI-generated --- two distributional regimes where standard evaluations offer little guidance.
arXiv:2606. 23701v1 Announce Type: cross Abstract: Qualitative product feedback can reveal nuanced user experiences, but its implicit sentiment is difficult to measure.
When people share experiences online, they often express thoughts in two ways: a star rating and a written review. In sentiment analysis, ratings are widely used as convenient weak labels for textual sentiment, yet whether the two actually agree is rarely questioned.
The study examines whether financial sentiment tools that are validated against human labels also reliably predict market outcomes. Using a large corpus of securities class action messages linked to abnormal stock returns, the authors compare five sentiment instruments—VADER, Loughran‑McDonald, FinBERT, Twitter‑RoBERTa, and an LLM annotator—within a single pipeline. Results show that the alignment between human agreement and sentiment scores varies with sampling strategy and time horizon: conventional sampling favors same‑day associations, while fixed‑n panels yield similar correlations for both same‑day and one‑day‑ahead predictions, yet overall predictive rankings remain weak.
arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.
arXiv:2608. 05183v1 Announce Type: cross Abstract: This dissertation analysed and discussed the differences in linguistic characteristics between pre-mortem and post-mortem social media content, and reported machine learning (ML) classifiers that achieved high performance in automatically detecting deaths of social networking site users from posts associated with their profiles.
arXiv:2502. 08266v3 Announce Type: replace-cross Abstract: Hate speech detection is a crucial task, especially on social media where harmful content can spread quickly.
arXiv:2609.36194v1 Announce Type: new Abstract: Extracted sentiment directions can vary across samples even when downstream sentiment classification remains accurate. To evaluate direction reproducib...
The paper examines whether human agreement and return association can be used interchangeably as criteria for validating sentiment tools in financial NLP. Using a large corpus of securities class action messages linked to abnormal stock returns, the authors compare five sentiment instruments and find that the relationship between human agreement and predictive validity varies with sampling conventions and score representations. They conclude that benchmark agreement establishes semantic validity but does not guarantee predictive rankings, and that message volume in a spam‑heavy conversation does not predict market damage or settlement size.
arXiv:2608. 03810v1 Announce Type: cross Abstract: Large language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events, and social groups, encoding affective framing alongside factual content: a target may appear favorable or threatening, calm or conflictual, powerful or vulnerable.
arXiv:2601.06631v2 Announce Type: replace Abstract: Building NLP systems for subjective tasks requires one to ensure their alignment to contrasting human values. We propose the MultiCalibrated Subjec...
arXiv:2607. 11890v1 Announce Type: cross Abstract: Open-ended surveys offer valuable insights, but they are notoriously difficult to analyze at scale.