← Back to all news
arXiv Computation and Language By Haruki Sakajo, Frederikus Hudi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

StructLens: A Structural Lens for Language Models via Maximum Spanning Trees

Read the original on arXiv Computation and Language →

StructLens is a framework that analyzes language model representations by constructing maximum spanning trees from semantic representations in residual streams, inspired by dependency parsing. It reveals that contiguous tokens are close in representation space, with middle layers showing the strongest local-span organization. The study also shows that smaller local units emerge earlier during pre‑training, while larger units appear later, offering insights into how token representations are organized across layers and training stages.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Sep 10

LLM Layers Immediately Correct Each Other

arXiv:2609.07876v1 Announce Type: cross Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...

By Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt
llmssafety
More like this →
arXiv AI
Jun 8

RePo: Language Models with Context Re-Positioning

arXiv:2512. 14391v3 Announce Type: replace-cross Abstract: In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices.

By Huayang Li, Tianyu Zhao, Deng Cai, Richard Sproat
llms
More like this →
arXiv AI
Sep 15

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
llmsragbenchmarks
More like this →
arXiv AI
Aug 11

Scaling Inherently Interpretable Language Models

arXiv:2608. 07594v1 Announce Type: cross Abstract: Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish.

By Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail, Giang Nguyen, Isaac Plant, Muawiz Chaudhary, Nathaniel Monson, Saqib Azim, Zhichen Guo, Julius Adebayo
llmsdiffusionsafety
More like this →