The paper introduces Jacobian Scopes, a set of gradient‑based, token‑level causal attribution techniques for large language models (LLMs). These methods, grounded in perturbation theory and information geometry, measure how individual input tokens affect specific logits, the overall predictive distribution, and model uncertainty. Experiments on instruction understanding, translation, and in‑context learning tasks show that Jacobian Scopes uncover political biases, translation strategies, and mechanisms in time‑series forecasting, and outperform existing attribution methods on token‑ablation benchmarks across six LLMs.
arXiv:2601. 16407v3 Announce Type: replace-cross Abstract: Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples.
The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.
The article examines whether large language models (LLMs) can effectively use representations they learn in-context to perform downstream tasks. Experiments with open‑weight LLMs on next‑token prediction and a new adaptive world modeling task reveal that these models struggle to deploy novel semantics defined in-context, even when such semantics are encoded in their latent representations. The study also shows that state‑of‑the‑art closed‑source reasoning models similarly fail to reliably leverage in‑context patterns, highlighting a gap in flexible representation usage.
arXiv:2508. 09904v3 Announce Type: replace-cross Abstract: Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form.
arXiv:2609.07876v1 Announce Type: cross Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...
The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.
arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.
The study demonstrates that large language models (LLMs) can perform numerical prediction on tabular data by leveraging semantic grounding—matching variable names to the values they describe—without task-specific training. Across 20 datasets from nine scientific domains, five LLMs matched or outperformed seven supervised machine learning models, especially when training examples were scarce. Controlled experiments with physical formulas reveal that LLMs combine domain knowledge (as a prior) with in-context examples (likelihood) in a Bayesian‑like manner, adjusting predictions as contradictory examples accumulate.
arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
arXiv:2609.28117v1 Announce Type: cross Abstract: In this paper, we introduce a gradient-based head attribution strategy where the Token-level Max-Margin loss is backpropagated to the attention maps....
arXiv:2609.17376v1 Announce Type: new Abstract: Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that supp...
Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood. We consider suc...