arXiv AI
Learning a Fact Is Not Learning How to Retrieve It
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 27
Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets
arXiv:2607. 21692v1 Announce Type: new Abstract: Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input.
arXiv Machine Learning
Jul 14
Can a Language Model Learn Facts Continually in Its Weights?
arXiv:2607. 11020v1 Announce Type: cross Abstract: Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights.
arXiv Machine Learning
Jul 30
Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
arXiv:2607. 21692v2 Announce Type: replace Abstract: Sparse attention prunes a long context to the blocks a model needs, and the usual selector is distilled from a dense teacher's attention.
arXiv AI
Aug 5
Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.
arXiv Machine Learning
Oct 8
When Should an In-Context Learner Expand Its Hypothesis Space?
The paper investigates when an in‑context learner should expand its hypothesis space, framing the decision as a costly sequential problem where prediction failures must be translated into structural evidence and then into an action. Using a Structural Revision Environment that allows exact Bayesian analysis, the authors find that the optimal decision to expand is governed by a value boundary rather than a simple evidence threshold, and that this boundary depends on the price of expansion, the remaining horizon, and the specific predictions that cannot be repaired locally. Experiments with transformers and language models show that while these models exhibit a failure‑sensitive signal, they often fail to incorporate cost and horizon considerations into their expansion decisions, and post‑training adjustments to priors or prediction sharpness do not alter the expansion criterion.
arXiv AI
Jun 11
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
arXiv:2606. 11198v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution.