← Back to all news
arXiv Computation and Language By Alexandra Kuleshova, Andrey Volodichev, Daria Kotova, Alexey Zaytsev

What LLMs Do in the Shadows: Reading Context Reliance from the Unembedding Dark Space

Read the original on arXiv Computation and Language →

The paper introduces DarkLens, a lightweight method for detecting faithfulness hallucinations in large language models (LLMs). It estimates context reliance by projecting hidden states onto the dark space—directions weakly aligned with the unembedding matrix—and uses the resulting Dark Matter Ratio (DMR) as a direct score of context utilization. Combined with a parametric knowledge score, DarkLens operates within a single forward pass and outperforms many baselines on QA and summarization benchmarks across several open‑weight LLMs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
llmsefficiency
More like this →
arXiv AI
Sep 15

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
llmsragbenchmarks
More like this →