arXiv:2510. 12071v2 Announce Type: replace Abstract: Current training data attribution (TDA) methods treat the influence one sample has on another as static, but neural networks learn in distinct stages that exhibit changing patterns of influence.
arXiv:2606. 04928v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance.
arXiv:2610.19041v1 Announce Type: new Abstract: Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve int...
arXiv:2609.24821v1 Announce Type: new Abstract: The Linear Representation Hypothesis associates high-level concepts with directions in language models, but it remains unclear how these concept-relate...
The Linear Representation Hypothesis associates high-level concepts with directions in language models, but it remains unclear how these concept-related linear structures are organized within the mode...
arXiv:2604. 16197v2 Announce Type: replace Abstract: Data attribution and valuation are critical for understanding data-model synergy for Large Language Models (LLMs), yet existing gradient-based methods suffer from scalability challenges on LLMs.
arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...
The paper investigates indirect evidence in language models by applying influence functions to identify training sentences that affect predictions on 15 grammatical phenomena from BLiMP. After filtering out direct evidence, the authors select the 60,000 most influential remaining sentences and show that removing them significantly degrades held‑out accuracy, more so than random removal. They further analyze the characteristics of this indirect evidence and demonstrate that clustering phenomena based on shared indirect evidence recovers known linguistic categories.
arXiv:2610.18520v1 Announce Type: cross Abstract: Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-tok...
arXiv:2601. 21996v2 Announce Type: replace-cross Abstract: While Mechanistic Interpretability has identified interpretable circuits in LLMs, their causal origins in training data remain elusive.
arXiv:2402. 08922v3 Announce Type: replace Abstract: Large-scale black-box models have become ubiquitous across numerous applications.
The paper investigates whether gradient‑based data attribution methods can filter out subliminal learning in language models. Three attribution techniques—GradCos, a contrastive variant, and EK‑FAC—are evaluated on three models and compared to divergence tokens, a strong baseline that requires counterfactual teacher models. Results show that EK‑FAC can mitigate some subliminal effects at the token level, but overall the methods are inconsistent and generally less effective than divergence tokens, especially when filtering entire samples.
Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training.