arXiv AI

Beyond Keywords: Leveraging Generative LLMs and Label Aggregation to Classify Economic Policy Uncertainty in News Articles

The paper presents a method that uses generative large language models (LLMs) to automatically classify news articles as discussing Economic Policy Uncertainty (EPU) and to identify the specific type of EPU. Unlike earlier keyword‑based or heavily supervised machine learning approaches, this work employs weak supervision by prompting LLMs to generate synthetic labels, making the process cost‑effective and scalable. It also introduces techniques for multi‑label and hierarchical classification of EPU‑related articles.

arXiv Machine Learning
Aug 24

MIL-BERT: Classification of Arbitrarily Large Text with Performance and Explanatory Guarantees

MIL-BERT is a neural network algorithm that classifies large texts by selecting relevant excerpts, inspired by multiple instance learning. It scales to samples with nearly 1 million tokens and has been evaluated on seven datasets, achieving state‑of‑the‑art results on three long‑text tasks such as political bias detection, trigger warning identification, and author demographic inference. The model also generalizes from weakly‑labeled text bags to accurately classify smaller instances.

By John Cadigan, Dayne Freitag, Eric Yeh
Hugging Face Trending Papers
Sep 2

Dutch Books for Language Models

The paper investigates the coherence of probabilistic forecasts produced by language models, particularly when users rely on them for life decisions involving uncertain events. Using a de Finetti-based method, the authors extract forecasts from language models about stock‑return events and compute the maximum Dutch‑book profit via linear programming, a metric of incoherence that does not require observed outcomes. The study finds significant incoherence, especially when events have complex logical relationships or when irrelevant context is present, and suggests that alternative training strategies could improve probabilistic coherence.

arXiv AI
Sep 3

Dutch Books for Language Models

The paper investigates the coherence of probabilistic forecasts produced by language models, particularly in the context of life‑decision support. Using a de Finetti‑based method, the authors elicit forecasts for events derived from stock return data and compute the maximum Dutch‑book profit via linear programming, which quantifies incoherence. They find significant incoherence, especially when events have complex logical relationships or when irrelevant context is present, and suggest that alternative training strategies could improve coherence.

By Isaiah Andrews, Suproteem Sarkar
arXiv AI
Sep 24

Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement

arXiv:2609.27165v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix backgr...

By Yuhe Wu, Rui Qian, Guangyu Wang, Yuran Chen, Yuanchao Zhu, Junjie Yang, Zhengheng Li, Jiulin Cai, Tianyi Zhang, Zihan Dong, Jiaxin Liu, Yujie Chen, Guang Zhang
arXiv AI
Sep 25

From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring

The paper explores using large language models (LLMs) as AI respondents to convert policy documents into structured survey responses. It introduces a long-context in‑context learning pipeline that maps policy text to predefined survey categories such as policy instruments, target groups, and thematic areas, and includes a secondary LLM validation step. Evaluation on a multi‑country dataset shows high agreement (84‑95%) with human responses for structured indicators, though free‑text fields differ, indicating that hybrid human‑AI workflows can enhance policy monitoring efficiency while still requiring human oversight.

By Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan, Arash Hajikhani
arXiv Machine Learning
Oct 16

A Large-Scale Dataset for Multi-Class Financial News Classification in Brazilian Portuguese

The paper introduces a large corpus of about 160,000 Brazilian Portuguese financial news articles to support Natural Language Processing research in this domain. It presents initial benchmarks for a weakly supervised multi‑class classification task using semantic similarity, evaluating cosine similarity, K‑Nearest Neighbors (KNN), and FAISS‑based nearest‑neighbor search with multilingual embeddings. Results show that KNN performs best on manually labeled data (47.3% accuracy, 31.9% F1), while LLM‑annotated data yields similar accuracy but much lower F1 scores due to label imbalance and noise.

By Leomax Bandeira, Gabriel Aguiar, Dalton C\'ezane Gomes Valadares, Luiz Pereira, Helder Fernando de Ara\'ujo Oliveira, Gustavo Xavier, Andson M. Balieiro
arXiv Computation and Language
2d ago

Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse

The paper investigates how to detect enthymemes—arguments with unstated premises or conclusions—in controversial political tweets about vaccination and immigration. Despite extensive fine‑tuning, models achieve an F1 score of only about 0.67, prompting the authors to propose new evaluation metrics that compare predictions to the full distribution of annotator votes and a model that directly predicts this distribution. The study also shows that models rely on explicit argumentative structure, often missing implicit links that readers must infer, raising concerns for content moderation systems that depend on majority labels.

By Martial Pastor, Nelleke Oostdijk