arXiv AI
The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 14
One mechanism for many mental spaces: a shared router over a value slot in language models
arXiv:2607. 10248v1 Announce Type: cross Abstract: Language builds discourse contexts other than the actual: a painting, a belief, a memory, a hypothetical.
arXiv Machine Learning
Jul 21
The Anatomy of a Truth Direction: Knowledge-Dependent Dimensionality, a Relational Law, and a Convergent Category Geometry in Small Language Models
arXiv:2607. 16741v1 Announce Type: new Abstract: B\"urger et al.
arXiv Machine Learning
Sep 2
How Do Language Models Choose Between Context and Memory?
The paper investigates how language models decide between contextual information and their internal memory when the two conflict. By estimating "authority directions" from agreement prompts and swapping these directions between matched prompts, the authors show that such interventions can reproduce 30–68% of the shift in source choice across Qwen, Llama, and OLMo models. Cross‑task experiments reveal that authority directions learned on one task transfer only modestly (≈9%) to another, indicating that authority computations are largely task‑specific.
arXiv AI
Sep 10
A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
A*-Thought-V2 is a framework that models Chain-of-Thought reasoning as a geometric trajectory in a 3D PCA space, using explicit-implicit latent tokens to compress steps that deviate from the main question-to-solution direction. The method measures alignment angles to decide which steps remain text and which become latent, and introduces stepwise embedding forcing and label forcing to train the architecture. Experiments on Qwen models show up to 2.6% accuracy gains, halved response length, and significant reductions in computation and training time.
arXiv Machine Learning
Jul 14
Can a Language Model Learn Facts Continually in Its Weights?
arXiv:2607. 11020v1 Announce Type: cross Abstract: Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights.
arXiv Machine Learning
Jul 15
Belief-reality separation lives in routing over a shared value slot in language models
arXiv:2607. 11945v1 Announce Type: cross Abstract: Capable language models hold what a character believes apart from what is true: told "Anna believes the cup is blue; in reality it is red," they answer blue about Anna and red about the world.