arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
arXiv:2608. 02050v1 Announce Type: cross Abstract: Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour?
Related stories
arXiv AI
Jun 2
DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models
arXiv:2606. 00091v1 Announce Type: cross Abstract: Joint Embedding Predictive Architectures (JEPAs) have reshaped self-supervised representation learning in vision.
Hugging Face Trending Papers
Jul 23
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap.
arXiv AI
Jul 24
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
arXiv:2607. 20594v1 Announce Type: cross Abstract: When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm?
arXiv AI
Jun 16
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
arXiv AI
Aug 13
Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
arXiv:2608. 11233v1 Announce Type: cross Abstract: A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing.
arXiv AI
Aug 18
Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation
arXiv:2608. 12385v2 Announce Type: replace Abstract: As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training.
arXiv Machine Learning
Jul 24
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
arXiv AI
Jun 2
ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks
arXiv:2606. 01080v1 Announce Type: cross Abstract: Large language models often improve on difficult tasks by spending inference-time compute on a reasoning trace before producing the final answer.
arXiv AI
Jul 28
Hierarchical Grading in Large Language Models
arXiv:2607. 22757v1 Announce Type: cross Abstract: We introduce Graded Large Language Models (GLLMs), an algebraic framework that equips the representation space of a transformer with a grading and propagates the induced weighted scalar action through embeddings, self-attention, and the training objective.
arXiv Computation and Language
Sep 25
Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters
The study re‑examines a reported advantage of a routed ternary (1.58‑bit) language model over a full‑precision transformer at 60K parameters. By running controlled experiments with multiple seeds and a fixed training recipe, the authors find that the apparent benefit largely stems from the choice of baseline model shape rather than the ternary architecture itself. While the routed model does outperform other shapes at a larger 130M‑byte budget, its advantage diminishes when a plain gated diagonal‑SSM block is used, and the ternary penalty varies with architecture and quantization details.
arXiv AI
Jun 9
CrossVLA: Cross-Paradigm Post-Training and Inference Optimization for Vision-Language-Action Models
arXiv:2605. 21854v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have rapidly converged on a small set of architectural patterns: discrete-token autoregression (e.