The paper investigates whether a global workspace—a set of verbalisable, causally potent representations—emerges in transformer models that use recurrence instead of a stack of distinct layers. Using a Jacobian lens extended with a virtual‑unrolling adapter, the authors analyze two recurrent transformer architectures, Ouro‑2.6B and Huginn‑0125, and compare them to a standard Qwen3.6‑27B baseline. They find that a workspace does form in the iterated parts of both models, but recurrence alters how it can be accessed: Ouro reconstructs workspace content in every loop and requires writes and ablations across all loops, whereas Huginn forwards content across all recurrences but limits reads, writes, and ablations to a sliding window of about two recurrences.
By Wenlong Wang, Fergal Reid
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
By Shubham Aggarwal
The paper introduces LOHA, a context layout that compresses older tool observations into soft tokens while keeping the agent’s own turns and the last K observations in plain text, and ACD, a training method that distills full‑text predictions into this latent representation while anchoring behavior on plain text. This approach reduces context per call by up to 57% without significant loss in resolve rates, and improves instance throughput in single‑GPU serving. Experiments on SWE‑bench Verified show that K=3 yields a 43–57% compression with only modest performance impact, while larger windows favor task performance over compression.
By Zhensheng Zou (Peking University), Guoqing Wang (Peking University), Dan Hao (Peking University)
Orthros introduces a phase‑aware heterogeneous attention mechanism that dynamically switches between linear‑complexity prefilling and full‑attention decoding, reducing the asymptotic cost compared to standard Transformers in long‑context scenarios. The method supports efficient Transformer‑to‑Orthros conversion, achieving higher performance than the original model and delivering a 21.9× prefilling speedup while maintaining strong in‑context recall, including 128K‑scale needle‑in‑a‑haystack tasks. These results demonstrate Orthros’s potential as a more efficient and flexible alternative for industrial Transformer applications.
By Yuhong Chou, Zehao Liu, Yuqi Pan, Qian Liu, Xianwei Chen, Zhongzhu Zhou, Jibin Wu
arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.
By Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis
arXiv:2610.08811v1 Announce Type: new
Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...
By Linfeng Dong
arXiv:2609.26219v1 Announce Type: cross
Abstract: Long-running LLM agent workflows often revise interior context spans while retaining long suffixes. Although suffix tokens remain unchanged, altered...
By Guotao Yang, Rui Guo, Siwei He, Sheng Chen, Yitao Hu, Keqiu Li
The Recurrent Looped Transformer (RLT) divides its layers into a parallel causal encoder and a recurrent decoder, allowing the computation path to grow with sequence length while keeping a fixed per-token cost. Experiments on six algorithmic tasks show that RLT outperforms a standard eight‑layer Transformer, achieving perfect generalization on parity up to 256 bits and high accuracy on swap‑based permutation tracking and modular arithmetic beyond training lengths. Ablation studies confirm that the recurrent feedback is essential for these gains, with chunked updates preserving performance on parity but reducing accuracy on permutation tracking.
By Yifan Zhang, Jichen Feng, Shihan Qin
arXiv:2607.05969v3 Announce Type: replace
Abstract: Latent memory, which stores past knowledge fragments as per-layer hidden states, has emerged as a promising paradigm (e.g., MemoryLLM and M+) for l...
By Ruiyi Yan, Zhuoyuan Mao, Yiwen Guo
arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.
By Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan
arXiv:2610.16301v1 Announce Type: new
Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...
By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)
The paper introduces "Trace as State," a method that places collected reasoning traces before a long-context block in a transformer, allowing earlier derived information to guide rereading. Experiments across three models and datasets show that this approach consistently outperforms a control where traces are appended after the context, achieving significant accuracy gains on tasks such as GraphWalks Parents and GLM-5.2. The results demonstrate that positioning traces before the context can enhance long-context reasoning while preserving the causal transformer architecture.
By Xu Zou, Jie Tang