The study investigates whether large language models retain traces of abandoned syntactic analyses after encountering garden‑path sentences. Across 24 models ranging from 124 M to 70 B parameters, the authors find that a residue of the initial, incorrect analysis influences later object attachment, while an adaptation toward the resolved structure scales with disambiguation error. Activation‑patching experiments show that the residue is carried by noun‑phrase states computed under the object analysis, and that overriding these states largely removes the effect.
By Xiang Ding
arXiv:2609.08372v2 Announce Type: replace
Abstract: A dense retriever encodes a question as one vector, but the question arrives one word at a time. We read 2,144 held-out headlines from Thu Vien Pha...
By Tran Minh Quan
The paper investigates how periodic subject changes—termed interruptions—affect the perceived novelty and coherence of text generated by base language models. By inserting a new subject every few hundred tokens into a stream that otherwise repeats, the authors find that judged surprise increases by 1.2 to 1.4 points and connection by 0.8 points compared to habituation alone. The study also reports that such interruptions do not produce integrated documents, and that the effect is robust across different models and evaluation protocols.
arXiv:2608. 19893v1 Announce Type: cross Abstract: Where does the novelty a base language model produces with no task come from, and what can an LLM judge of a long stream actually see?
By Roberto I. Ono Filho
What do a language model's hidden states say about the organization of a single text? From one forward pass, without training, we score every token position on two properties.
arXiv:2608. 09093v1 Announce Type: cross Abstract: How a document's arrangement is written down, its notation, is a training variable that no dataset card records.
By E. M. Freeburg
The paper studies how transformer representations evolve across layers by examining the intrinsic dimensionality (ID) of token embeddings and their neighborhood structures. It finds that closed‑class tokens expand and collapse earlier than open‑class tokens, and that these changes are linked to shifts in local geometry. The authors compare encoder and decoder models, showing distinct layer‑wise behaviors, and demonstrate that geometric features alone can predict a token’s part‑of‑speech and reveal how semantic content changes across layers.
By Samuele Vallisa, Federico Ravenda, Claudio Palominos, Rui He, Andrea Raballo, Antonietta Mira, Philipp Homan, Wolfram Hinzen
The paper investigates neural text degeneration by measuring the fixed‑point structure of short‑window argmax maps across 17 pretrained models, using 96 random two‑token starts without prompts. It finds a stable four‑way classification that varies across model families and scales, with some models funneling to a single endpoint token while others do not, and shows that this behavior is not solely determined by training data or corpus frequency. The study demonstrates that repetition phenomena are not uniformly explained by either training data or network architecture alone, highlighting the complexity of neural text generation dynamics.
By Nicol\'as Vera Z\'u\~niga
The paper argues that language operates with two parameters: amplitude, which measures how often words co‑occur, and phase, a signed relational factor that determines how co‑activated meanings combine and can reverse a meaning’s contribution. Unlike amplitude, phase is not captured by standard word embeddings or transformer attention weights and is indexed to individuals and dyadic interactions. The authors propose six empirical predictions to test phase’s role and suggest that future language models should incorporate agent‑indexed, phase‑bearing semantic states.
The paper investigates whether language models can identify sentences from their training data by using exact duplication counts from publicly released corpora for two model families, OLMo‑2 and Pythia. It finds that for typical duplication levels, models show only a weak trace of exposure, with a rank correlation near –0.08, and that strong signals only appear when a sentence appears roughly a thousand times, at which point fame rather than memory dominates. The study also demonstrates that common membership tests can be misleading, as changing a single word does not alter the model’s preference, and that controlling for register can significantly improve detector performance.
By Arman Nik Khah
arXiv:2608. 04021v1 Announce Type: cross Abstract: Cloze-style probes that vary how often a target token appears implicitly assume that more copies of a target affect prediction the same way regardless of where the readout slot sits.
By Han-yu Wang
The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.
By Micah Adler, John W. Byers, Mark Crovella