arXiv Machine Learning

Information-Matched Auditing of Conversational Emotion-Shift Forecasting: A Leakage-Safe Evaluation Protocol

The paper introduces an information‑matched, leakage‑safe auditing protocol for evaluating conversational emotion‑shift forecasting models. It addresses common distortions such as unequal feature richness, noisy current‑emotion labels, and differing hyperparameter budgets by using explicit commitment points, causality tests, balanced configuration searches, and seed‑dialogue‑aware inference. Applying the protocol to six causal model families across four corpora shows that a deployable train‑only current‑emotion classifier improves over a predicted‑label baseline, especially on DailyDialog, while matched‑gold conditions reveal baseline competitiveness on IEMOCAP and stronger performance on MELD.

arXiv Computation and Language
Nov 5

Selective, Risk-Controlled Emotion Forecasting in Conversation: From a Persistence Barrier to Departure Detection with Conformal Guarantees

The paper redefines emotion forecasting in conversation by moving away from predicting continuous emotional values toward a selective, risk‑controlled departure detection framework. Experiments on valence‑arousal‑dominance trajectories show that simple persistence baselines outperform complex frozen‑encoder models, and that the key predictive signal is the timing of a departure from persistence. A dialogue‑level conformal risk control mechanism is introduced to bound expected loss, achieving target guarantees across twelve configurations, though coverage drops on the most challenging corpus.

By Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan
arXiv AI
Sep 2

Some Emotions Run Deeper: Layer-wise Probing and Causal Intervention in Large Language Models

The study examines how emotions are represented across layers of large language models (LLMs) by probing eight 1B–9B open‑weight models on three datasets (Twitter, Reddit, autobiographical narratives). It finds that the optimal probing layer varies systematically with the dataset, moving from near‑input layers to deeper layers, and that targeted forward‑pass interventions on these layers degrade performance more than random interventions. Additionally, the selected layers transfer across datasets and emotion categories, and early‑exit representations from these layers outperform full‑depth exits by an average of 6.9 percentage points.

By Tian Fang, Ga\"el Guibon, Davide Buscaldi
arXiv AI
Aug 24

Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding

The paper investigates how streaming emotion recognition models can be misled by their own prior predictions, a problem termed previous-belief contamination (PBC). Using a counterfactual diagnostic on CREMA-D-Stream, the authors show that feeding a model’s previous emotion label into its current prediction can drastically lower accuracy and flip many predictions, with the effect varying by label. To mitigate PBC, they propose EmoUpdate, a training‑free framework that isolates current audio perception from historical context through a prior‑blind firewall, a causal belief filter, and a decontamination operator, achieving significant gains across multiple SpeechLMs and benchmarks.

By Haoyue Liu, Zhichao Wang, Ye Chen, Haonan Deng, Xiaoying Tang
arXiv AI
Sep 30

Alignment Forecasting: Predicting Misalignment From Training Data

The paper introduces Alignment Forecasting, a method for predicting whether fine‑tuning a language model on a given dataset will increase specific alignment failures such as deception or sycophancy. It presents ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions across many models, datasets, and failure modes, and shows that a simple forecasting scaffold using an LLM’s assessment of dataset bias can outperform baseline forecasters. The authors demonstrate that filtering out high‑risk training examples identified by the forecaster can improve alignment in multiple‑choice evaluations, though benefits in open‑ended conversations remain uncertain.

By Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky, Tomek Korbak
arXiv Machine Learning
6d ago

Auditing the Judge: Internal Signals for LLM Reasoning Assurance

The paper investigates how internal signals from large language models can be used to improve error detection and budget allocation for answer review. By training a supervised correctness probe on attention-head activations, the authors show that this probe can identify 1.3–22.5 percentage points more errors at a 5% review budget across several 7–12B models on GSM8K and MATH, outperforming the model’s own self-judgment in many settings. While the probe improves in-domain allocation, its effectiveness drops when transferred across datasets, indicating that internal signals are not universally reliable confidence estimators.

By Mayank Gulati, Gerhard Wunder
arXiv AI
Nov 5

Mass Producing Training Data with Surprising Side Effects

The paper "Mass Producing Training Data with Surprising Side Effects" presents a method for automatically discovering datasets that cause unexpected side effects in language models. By defining three criteria—target behavior elicitation, ordinary appearance, and human surprise—the authors use LLM judges and search techniques to generate 6,575 datasets across eight behaviors. These datasets, which generalize across model families and sizes, provide a testbed for studying generalization and predicting training side effects before deployment.

By Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann, Cassidy Laidlaw
arXiv AI
Sep 23

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

arXiv:2609.24677v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to make predictions from numerical time-series histories and textual events. Yet accuracy alone cann...

By Jie Gong, Maowei Jiang, Zhiwei Liu, Yankai Chen, Guojun Xiong, Xue Liu, Min Peng, Qianqian Xie, Sophia Ananiadou