The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2608. 12334v1 Announce Type: cross Abstract: Despite the impressive multilingual capabilities of Large Language Models, the latent dynamics dictating language selection remain poorly understood.
arXiv:2607. 12279v1 Announce Type: cross Abstract: Writing a sentence of exactly twelve words; ending a DNA sequence at the right codon; formatting an ASCII table.
Writing a sentence of exactly twelve words; ending a DNA sequence at the right codon; formatting an ASCII table. These are all tasks that language models can do that requires tracking how many tokens remain before a target.
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv:2603.29396v2 Announce Type: replace Abstract: Standard evaluations of Large language models (LLMs) focus on task performance, offering limited insight into whether correct behavior reflects app...
The paper introduces a method to embed a controllable latent variable into natural-looking text by steering a teacher LLM along eight sparse autoencoder directions that follow a ring-shaped Markov chain. A small transformer trained on this data successfully tracks the Bayesian posterior of the planted variable and arranges the eight states in the same ring order, linking belief states to concept geometry. This demonstrates that LLMs can model latent variables and that concept geometry may arise from the statistical dynamics of these variables.