arXiv AI September 15, 2026 By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

Read the original on arXiv AI →

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Sep 10

LLM Layers Immediately Correct Each Other

arXiv:2609.07876v1 Announce Type: cross Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...

By Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt

llms safety

arXiv AI

Aug 24

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

MGAL is a new multilingual benchmark for evaluating long‑context large language models, built from United Nations reports in six official UN languages and covering 8K to 128K tokens. It tests four linguistic granularities—word, sentence, paragraph, and document—while also stratifying examples by their position within the document (begin, middle, end). Experiments show that models excel at word‑level tasks but struggle with coarser granularity, and that closed‑source models outperform others in lower‑resource languages, revealing challenges such as local semantic crowding and a fluency‑consistency gap.

By Chunhan Li, Chenglin Xu, Zongyang Zhang, Jiale Liu, Zhuoxi Rao, Xudong Jia, Junxiu He, Menglin Yang, Wenjuan Gong, Zhengzhe Liu, Chengwei Qin

llms benchmarks

arXiv AI

Jun 12

LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

arXiv:2602. 00462v4 Announce Type: replace-cross Abstract: Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM.

By Benno Krojer, Shravan Nayak, Oscar Ma\~nas, Vaibhav Adlakha, Desmond Elliott, Siva Reddy, Marius Mosbach

llms rag multimodal safety

arXiv AI

Jun 2

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

arXiv:2606. 00909v1 Announce Type: cross Abstract: This work presents MLLM-Microscope, a novel system designed for analyzing the hidden representations within Multimodal Large Language Models (MLLMs).

By Ravil Mussabayev, Rustam Mussabayev

llms rag multimodal benchmarks

arXiv AI

Jun 9

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

arXiv:2606. 07604v1 Announce Type: cross Abstract: Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs).

By Harry Jake Cunningham, Nicola Muca Cirone

llms safety

arXiv AI

Jul 14

Extending LLM Context via Associative Recurrent Memory

arXiv:2607. 11614v1 Announce Type: cross Abstract: Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling.

By Gleb Kuzmin, Ivan Rodkin, Aydar Bulatov, Yuri Kuratov, Lyudmila Rvanova, Mikhail Katkov, Ilia Sochenkov, Misha Tsodyks, Timothy Baldwin, Mikhail Burtsev, Artem Shelmanov

llms fine-tuning

One email a morning, machine-written

Related stories

LLM Layers Immediately Correct Each Other

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Extending LLM Context via Associative Recurrent Memory