arXiv:2607. 21692v1 Announce Type: new Abstract: Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input.
arXiv:2607. 11020v1 Announce Type: cross Abstract: Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights.
arXiv:2607. 21692v2 Announce Type: replace Abstract: Sparse attention prunes a long context to the blocks a model needs, and the usual selector is distilled from a dense teacher's attention.
arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.
The paper investigates when an in‑context learner should expand its hypothesis space, framing the decision as a costly sequential problem where prediction failures must be translated into structural evidence and then into an action. Using a Structural Revision Environment that allows exact Bayesian analysis, the authors find that the optimal decision to expand is governed by a value boundary rather than a simple evidence threshold, and that this boundary depends on the price of expansion, the remaining horizon, and the specific predictions that cannot be repaired locally. Experiments with transformers and language models show that while these models exhibit a failure‑sensitive signal, they often fail to incorporate cost and horizon considerations into their expansion decisions, and post‑training adjustments to priors or prediction sharpness do not alter the expansion criterion.
arXiv:2606. 11198v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution.
arXiv:2610.27580v1 Announce Type: cross Abstract: Factual recall is a lookup: some representation inside the network serves as the key, and what that key is made of decides which queries can retrieve...
The paper investigates how large language models (LLMs) encode and retrieve facts based on their order of mention in a context. By measuring changes in hidden states when shifting questions from one fact to another, the authors identify an "ordinal vector" that points to a fact’s position in the list rather than its name. Across several models (Qwen, Gemma, Llama), these ordinal addresses are ordered, steerable, low‑rank, and emerge in mid‑layers, suggesting that LLMs locate facts by their mention order, mirroring aspects of human recall.
arXiv:2507. 01414v2 Announce Type: replace Abstract: We introduce a new family of toy problems that combine features of linear-regression-style continuous in-context learning (ICL) with discrete associative recall.
arXiv:2505.09338v3 Announce Type: replace Abstract: We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanisti...
arXiv:2605.30251v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often solve a task when all instructions are given in a single prompt, but fail when the same information is rev...
arXiv:2606. 31495v1 Announce Type: new Abstract: We study a single idea across two settings: that a prediction-error signal, computed by a small predictor over the latent space of a frozen encoder, can serve both as a gate on plasticity and as a substrate for metacognition.