The paper proposes Bypass Observation, a non‑intrusive layer‑wise readout architecture that attaches read‑only observation heads to selected Transformer layers without feeding their outputs back into the backbone. Three variants are explored: a shared language‑model head across layers, layer‑specific heads, and a layer‑ or step‑adaptive head. The authors provide a closed‑form overhead estimate (≈ V/(12d)) and discuss ways to reduce cost, while distinguishing bypass chain‑of‑thought from conventional chain‑of‑thought and outlining potential applications to looped and recurrent‑depth Transformers.
By Haibin Tong, Jiang Yu
arXiv:2603.21676v2 Announce Type: replace-cross
Abstract: Standard Transformers have a fixed computational depth, limiting their ability to generalize to tasks that require variable-depth reasoning....
By Hung-Hsuan Chen
arXiv:2607. 22925v1 Announce Type: cross Abstract: A key question for AI safety is whether a language model expresses all of its reasoning in its output tokens.
By Vatsal Baherwani, Tom Goldstein, Ashwinee Panda
The paper investigates whether extra computation in recurrent Transformers should be allocated to more temporal steps or greater physical depth. Using Latent Recurrent Transformers (LRTs), the authors insert a latent thought token between vocabulary tokens, allowing each token to pass through the same $L$ layers twice while sharing parameters. Experiments on 16‑ and 20‑layer mixture‑of‑experts NanoChat backbones show that a single thought token brings a shallower model within 0.006–0.004 bits per byte of a double‑depth counterpart, recovering 67–81% of the improvement with roughly 48% fewer parameters.
By Zeyi Huang, Xuehai He, Yong Jae Lee, Yelong Shen
arXiv:2609.16372v1 Announce Type: new
Abstract: Masked diffusion language models (dLLMs) generate text by iteratively denoising masked tokens with bidirectional attention. Extending reasoning across...
By Albert Ge, Chandan Singh, Yufan Zhuang, Xiaodong Liu, Jianfeng Gao, Frederic Sala
arXiv:2606. 01080v1 Announce Type: cross Abstract: Large language models often improve on difficult tasks by spending inference-time compute on a reasoning trace before producing the final answer.
By Dhruv Saini, Rohan Pandey