arXiv AI
FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 16
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
arXiv AI
Aug 11
ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory
arXiv:2606. 25156v3 Announce Type: replace-cross Abstract: Native length extrapolation remain a weakly solvable problem in language modeling due to trade-off balancing between exact retrieval fidelity, long-document likelihood, and inference efficiency.
arXiv Computation and Language
2d ago
Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
arXiv AI
Oct 2
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
MWOP (Modality-aware Width-wise Operation Pruning) is a method that independently prunes visual‑to‑visual, text‑to‑visual, and text‑to‑text attention paths within each layer of multimodal large language models, and separately selects feed‑forward network channels for visual and textual inputs. It uses a first‑order Taylor criterion to guide pruning, re‑evaluates FFN importance after attention pruning, and applies LoRA‑based recovery training. The approach is paired with path‑sparse Triton attention kernels and compact visual‑side FFN execution to achieve practical acceleration, preserving token sequences while reducing computation. "whyItMatters":"MWOP achieves a 1.6× prefill speedup on LLaVA‑OneVision‑7B while retaining 99.7% performance, and further boosts token‑compression methods to 2.9× and 2.7× speedups, demonstrating its effectiveness across architectures."
arXiv AI
Jul 10
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
arXiv:2607. 07953v1 Announce Type: cross Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context.
arXiv AI
5d ago
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Orthros introduces a phase‑aware heterogeneous attention mechanism that dynamically switches between linear‑complexity prefilling and full‑attention decoding, reducing the asymptotic cost compared to standard Transformers in long‑context scenarios. The method supports efficient Transformer‑to‑Orthros conversion, achieving higher performance than the original model and delivering a 21.9× prefilling speedup while maintaining strong in‑context recall, including 128K‑scale needle‑in‑a‑haystack tasks. These results demonstrate Orthros’s potential as a more efficient and flexible alternative for industrial Transformer applications.