arXiv Machine Learning

MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecasting

MemGuard-Alpha evaluates whether membership inference attacks (MIA) can detect memorization in large language models (LLMs) used for financial alpha signals. The study combines five MIA methods with a temporal proximity feature and a cross-model disagreement metric, then audits them across seven LLMs, 50 S&P 100 stocks, and 299,600 prompt-model pairs. Findings show that temporal proximity alone perfectly predicts in-sample status, MIA discriminative power largely stems from model scale differences, and filtering based on contamination scores does not improve risk-adjusted performance once transaction costs are considered.

arXiv Machine Learning

Aug 19

Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal

The paper audits the impact of temporal leakage on financial-news direction prediction across 49,799 articles and 16 feature-model combinations, including TF‑IDF, MiniLM, FinBERT, and fine‑tuned RoBERTa‑large / DeBERTa‑v3‑large, as well as zero/few‑shot and LoRA probes of Llama‑3 and Qwen2.5. Random train‑test splits inflate MCC scores by 1.1× to 6.5×, with larger models and richer features showing greater gains, while end‑to‑end FinBERT fine‑tuning actually increases the gap. Only the mergers and acquisitions (M&A) category shows a positive locked‑test signal under near‑temporal chronological evaluation, with the signal localized to 2024‑2025 European‑tilted M&A semantics and not transferring to a 2009‑2020 U.S. corpus.

By Chenhao Xue, Raslen Guesmi, Siwei Feng, Yucheng Gong, Jacob Xavier Sundram, Jordan Pang, Lan Wang, Julian Kaljuvee

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Aug 11

Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

arXiv:2608. 08126v1 Announce Type: new Abstract: Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters, in tension with supervisory expectations that adverse decisions be explainable.

By Gregorius Reynaldi Pratama, Kuo-Kun Tseng

llms

arXiv AI

Sep 1

Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models

arXiv:2605.24564v2 Announce Type: replace Abstract: Backtesting large language models (LLMs) on historical financial data is unreliable when their pre-training data include the evaluated events. An L...

By Weixian Waylon Li, Mengyu Wang, Tiejun Ma

llms benchmarks safety

arXiv Computation and Language

Sep 4

Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards

The paper investigates how benchmark contamination—leakage of test items into training data—affects large language model (LLM) leaderboards. By comparing original test items with semantically equivalent paraphrases, the authors measure contamination as a violation of anchor-item invariance and find that it inflates absolute scores but rarely changes model rankings. Across 47 public models and 74 finetuned models on four benchmarks, the rank correlation between standard and paraphrase-controlled leaderboards is 0.997, with only a handful of cases showing differential contamination that could alter rankings.

By Xingyao Xiao (Stanford University), Yihong Cheng (City University of Macau)

llms benchmarks

arXiv AI

Sep 15

MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents

arXiv:2609.14976v1 Announce Type: new Abstract: Long-horizon LLM agents accumulate memory across sessions, creating sparse but high-impact risks: stale facts, conflicting updates, cross-user leakage,...

By Jianhua Jiang, Dongbo Yuan, Weihua Li

llms agents efficiency benchmarks

Hugging Face Trending Papers

Sep 9

A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out

Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious...

benchmarks

arXiv Machine Learning

Jul 15

Extractable Memorization From First Principles

arXiv:2607. 12649v1 Announce Type: new Abstract: Recent work on extractable memorization in LLMs suffers from two contrasting validity problems.

By A. Feder Cooper, Marika Swanberg, Jamie Hayes, Lea Duesterwald, Christopher De Sa, Daniel E. Ho, Mark A. Lemley, Percy Liang

llms

arXiv AI

Aug 17

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.

By Zhelun Wu

llms

arXiv Machine Learning

Jul 21

Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation

arXiv:2604. 16197v2 Announce Type: replace Abstract: Data attribution and valuation are critical for understanding data-model synergy for Large Language Models (LLMs), yet existing gradient-based methods suffer from scalability challenges on LLMs.

By Yide Ran, Jianwen Xie, Minghui Wang, Wenjin Zheng, Denghui Zhang, Chuan Li, Zhaozhuo Xu

llms

arXiv Machine Learning

Aug 18

MoRFI: Monotonic Sparse Autoencoder Feature Identification

arXiv:2604. 26866v2 Announce Type: replace-cross Abstract: Large language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next token prediction.

By Dimitris Dimakopoulos, Shay B. Cohen, Ioannis Konstas

llms fine-tuning safety

arXiv Machine Learning

Aug 12

Leak It: Per-Document Extraction Beyond Aggregate Membership Inference

arXiv:2608. 00144v2 Announce Type: replace Abstract: Membership inference (MIA) on language models is usually summarised by aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines can separate members from non-members using surface text alone.

By Victor Maricato

llms safety

arXiv Machine Learning

Sep 1

When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction

arXiv:2608.30086v1 Announce Type: cross Abstract: Credit-default prediction is an important task in financial decision making. Traditional methods use fitted classifiers such as logistic regression a...

By Rishi Datta, Lavanya Prahallad

llms efficiency

Related stories