← Back to all news
arXiv AI By Alexandre Quemy

Function Lives Where Variance Doesn't: Task-Weighted Charts of a Language Model's Computation

Read the original on arXiv AI →

The paper introduces task‑weighted charts, a method that defines low‑dimensional coordinate systems based on a chosen functional of a language model’s representation. Using these charts, the authors show that next‑token prediction requires 70–90% of the residual stream’s width to maintain perplexity, a width largely unused by variance‑based analyses. The study demonstrates that charts trained under the functional’s metric preserve predictions better than variance‑based or optimal linear compression when only a few dimensions are retained.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 10

Limits of Reliability and Scaling in Language Models

The paper argues that large language models cannot achieve perfect reliability for any task, even with unlimited scale. It establishes that each generative task has an inherent reliability ceiling set by how much output uncertainty can be resolved from observable context, with a resolvable part that can be improved by more context and a subjective part tied to task ambiguity. The authors derive a scaling law showing that performance is limited by the scarcer resource—either training data or model capacity—and explain how this law explains phenomena such as retrieval augmentation and catastrophic forgetting.

By Subhabrata Majumdar
llms
More like this →
arXiv Machine Learning
Jul 15

Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

arXiv:2603. 06592v2 Announce Type: replace-cross Abstract: Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect.

By Jonas Rohweder, Subhabrata Dutta, Iryna Gurevych
llmssafety
More like this →