arXiv Machine Learning August 6, 2026 By George Fountzoulas

Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

Read the original on arXiv Machine Learning →

arXiv:2608. 04678v1 Announce Type: cross Abstract: Papers 1-2 of the Kathleen series showed that a byte-level, attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 17

Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect

arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?

By Ely Hahami, Ishaan Sinha, Lavik Jain

llms fine-tuning benchmarks safety

arXiv AI

Aug 11

ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory

arXiv:2606. 25156v3 Announce Type: replace-cross Abstract: Native length extrapolation remain a weakly solvable problem in language modeling due to trade-off balancing between exact retrieval fidelity, long-document likelihood, and inference efficiency.

By Habibullah Akbar

benchmarks

Hugging Face Trending Papers

Jun 21

Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction

Large language models (LLMs) achieve strong relation extraction (RE), but their computational demands and reliance on proprietary APIs limit deployment in resource-constrained or privacy-sensitive settings. We investigate how far small language models (SLMs) can close this gap across general-domain and literary text.

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 9

FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

arXiv:2607. 07478v1 Announce Type: new Abstract: FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling.

By Athanasios Zeris

llms rag

arXiv Machine Learning

Jul 27

RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

arXiv:2607. 21927v1 Announce Type: new Abstract: Full self-attention in large language models scales as O(N^2), which limits long-context document analysis to 65,536 tokens and requires costly GPU clusters.

By Anderson R. Santos

llms

arXiv AI

Jun 16

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing

arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.

By Kuzey Torlak, H\"useyin Arda Arslan, An{\i}l Dervi\c{s}o\u{g}lu, Beyza Nur Deniz, Onur Boyar

llms nlp efficiency

Related stories

Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect

ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory

Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction

FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing