arXiv AI

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

arXiv Computation and Language
Nov 4

FinForesight: A Diagnostic Backtesting Benchmark for Deep Research Systems in Financial Forecasting

FinForesight is a diagnostic backtesting benchmark designed to evaluate deep research systems for financial forecasting. It tests four forecasting tasks—valuation, return ranking, event direction, and future financial fundamentals—across Chinese, U.S., and Hong Kong markets, while also including two diagnostic tasks for counterfactual reasoning and financial calculation. The framework limits evidence to what was available at the forecast date and compares pretraining-era, post-cutoff, and live evaluations to assess how well controlled backtests predict future performance.

By Fengyuan Liu, Mukai Li, Jieshun You, Yilin Zheng, Chenjia Min, Yizhou Zhang, Harper Chen, Qi Liu
arXiv AI
6d ago

Predictive Surprise as Self-Grounding Concept Bottleneck for Interpretable Time Series

The paper introduces ConceptTime, a time‑series classifier that achieves accuracy, interpretability, and abstention by using a self‑supervised predictive surprise signal. A frozen probabilistic forecaster generates predictions at each timestep; the mismatch between prediction and actual observation identifies segment boundaries and defines concepts that describe each segment’s dynamics. ConceptTime’s lightweight head classifies from these concepts, and the distance to the nearest concept provides a built‑in reliability measure, achieving state‑of‑the‑art performance across 17 benchmarks without requiring annotations or expert input.

By Sachith Abeywickrama, Emadeldeen Eldele, Min Wu, Xiaoli Li, Chau Yuen
arXiv Machine Learning
Nov 5

Information-Matched Auditing of Conversational Emotion-Shift Forecasting: A Leakage-Safe Evaluation Protocol

The paper introduces an information‑matched, leakage‑safe auditing protocol for evaluating conversational emotion‑shift forecasting models. It addresses common distortions such as unequal feature richness, noisy current‑emotion labels, and differing hyperparameter budgets by using explicit commitment points, causality tests, balanced configuration searches, and seed‑dialogue‑aware inference. Applying the protocol to six causal model families across four corpora shows that a deployable train‑only current‑emotion classifier improves over a predicted‑label baseline, especially on DailyDialog, while matched‑gold conditions reveal baseline competitiveness on IEMOCAP and stronger performance on MELD.

By Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan
arXiv AI
Oct 26

Do Language Models Trust Their Own Justifications? A Study on Functional Consistency

The paper investigates whether the tokens highlighted by large language models (LLMs) as key to their predictions are behaviorally consistent when those tokens are manipulated. Using sufficiency, comprehensiveness, and counterfactuality metrics, the authors evaluate closed‑source GPT‑4o‑mini and open‑source models Gemma3, Granite8B, and DeepSeek on IMDB and Steam review datasets. Results show that GPT‑4o‑mini consistently follows expected patterns, Gemma3 and Granite8B maintain sufficiency but falter under stricter interventions, and DeepSeek variants exhibit structural inconsistencies, indicating that explanation reliability varies across model families and sizes.

By Alisson Rosa Pereira, Bruno Iochins Grisci