← Back to all news
arXiv AI By Jung H. Lee, Sujith Vijayan

Functional Subspace, where language models can use vector algebra to solve problems

Read the original on arXiv AI →

The paper proposes that large language models (LLMs) encode high‑level concepts as linear directions within their activation space and that they can use subspaces and vector algebra to perform tasks. By analyzing functional modules and residual streams during in‑context learning (ICL), the authors find that LLMs can create evidence‑accumulating subspaces and solve ICL tasks through simple algebraic operations within those subspaces.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 15

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
llmsragbenchmarks
More like this →
arXiv AI
Sep 21

Understanding In-context Learning of Addition via Activation Subspaces

The paper investigates how transformer language models perform few‑shot learning for a simple addition task, showing that the ability is concentrated in a handful of attention heads. Using dimensionality reduction, the authors identify low‑dimensional subspaces—three heads with six‑dimensional spaces in Llama‑3‑8B‑Instruct—where specific dimensions encode the units digit via trigonometric patterns and magnitude via low‑frequency components. They also derive a mathematical identity linking aggregator and extractor subspaces, enabling tracking of information flow from examples to the final prediction.

By Xinyan Hu, Kayo Yin, Michael I. Jordan, Jacob Steinhardt, Lijie Chen
llms
More like this →
arXiv Machine Learning
4d ago

Are In-Context Images Worth 10 Dimensions?

arXiv:2609.37659v1 Announce Type: cross Abstract: There has been significant work on understanding the In-Context Learning capabilities of Large Language Models, especially on the induction circuit....

By Adhemar de Senneville, Xavier Bou, J\'er\'emy Anger, Rafael Grompone, Gabriele Facciolo
llmscomputer-vision
More like this →