← Back to all news
arXiv AI By Grayson Wycliffe Storer, Julia Witte Zimmerman

Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction

Read the original on arXiv AI →

The paper introduces token‑wise incremental trajectories, a method that tracks how contextual word embeddings evolve as each word is added to a sentence. Using garden‑path sentences, the authors show that these trajectories capture known processing disruptions and can distinguish ambiguous sentences from disambiguated controls. They propose metrics for measuring representational displacement and find that ambiguity signals are present not only in sentence‑level CLS vectors but also in individual token embeddings, suggesting distributed meaning construction across scales.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 15

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
llmsragbenchmarks
More like this →
arXiv Computation and Language
6d ago

Coupled Usage-Sense Processes: Temporal and Attributable Lexical Semantic Change

The paper introduces Coupled Usage–Sense Processes (CUSP), a method that models lexical semantic change by coupling contextual distributions through latent usage components and using Markov composition to link adjacent time periods. CUSP quantifies change magnitude and timing, separates variation into component movement and internal reorganization, and attributes changes to specific transported component pairs. The approach is validated on synthetic data, English and German corpora, and a large corpus of US court opinions, providing detailed, text‑grounded insights into how word meanings evolve over time.

By Haruka Ezoe, Ryohei Hisano
More like this →
arXiv Computation and Language
Aug 27

The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

The paper studies how transformer representations evolve across layers by examining the intrinsic dimensionality (ID) of token embeddings and their neighborhood structures. It finds that closed‑class tokens expand and collapse earlier than open‑class tokens, and that these changes are linked to shifts in local geometry. The authors compare encoder and decoder models, showing distinct layer‑wise behaviors, and demonstrate that geometric features alone can predict a token’s part‑of‑speech and reveal how semantic content changes across layers.

By Samuele Vallisa, Federico Ravenda, Claudio Palominos, Rui He, Andrea Raballo, Antonietta Mira, Philipp Homan, Wolfram Hinzen
llmsnlp
More like this →