arXiv Machine Learning August 6, 2026 By Yang Bai, Linyuan Wang, Haoyang Jiang, Nuolin Sun, Libin Hou, Bin Yan

Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining

Read the original on arXiv Machine Learning →

arXiv:2608. 04213v1 Announce Type: new Abstract: Existing studies on self-supervised learning for white-box networks typically decouple the derivation of white-box networks via optimization algorithms from self-supervised learning paradigms.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Sep 15

Disentangling Representation Evolution in Transformers through Directional Decomposition

arXiv:2609.15975v1 Announce Type: cross Abstract: Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study thi...

By Shwai He, Haichao Zhang, Shen Yan

llms robotics

arXiv Machine Learning

Jul 13

Training, Reading, and Editing Legible Transformers

arXiv:2607. 08946v1 Announce Type: new Abstract: A transformer can be built from operators that are legible by construction -- bounded, named units that read as fuzzy set operations rather than dense activations -- but legibility must be pressed for during training, and the pressure has a failure mode.

By Mark Oskin

llms

Hugging Face Trending Papers

3d ago

Learning Functional Subspaces for Neural Network Compression

Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standar...

llms efficiency

arXiv Machine Learning

4d ago

Self-Supervised Representation Learning as Mutual Information Maximization

arXiv:2510.01345v2 Announce Type: replace Abstract: Self-supervised representation learning (SSRL) has demonstrated remarkable empirical success, yet its underlying principles remain insufficiently u...

By Akhlaqur Rahman Sabby, Yi Sui, Tongzi Wu, Jesse C. Cresswell, Ga Wu

efficiency

arXiv Computation and Language

3d ago

Learning Functional Subspaces for Neural Network Compression

arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...

By Massimo Bini, Anders Christensen, Stephan Alaniz, Judah Goldfeder, Ole Winther, Yann LeCun, Ravid Shwartz-Ziv, Zeynep Akata

llms efficiency

arXiv Machine Learning

Jun 16

Transformers Learn the Mestre-Nagao Heuristic

arXiv:2606. 15036v1 Announce Type: new Abstract: We train a two-layer transformer encoder to classify rational elliptic curves $E/\mathbb{Q}$ of conductor $\leq 10000$ as either rank 0 or rank 1 from the first 128 normalized Frobenius traces.

By Pranav Venkata Konda

llms rag safety

One email a morning, machine-written

Related stories

Disentangling Representation Evolution in Transformers through Directional Decomposition

Training, Reading, and Editing Legible Transformers

Learning Functional Subspaces for Neural Network Compression

Self-Supervised Representation Learning as Mutual Information Maximization

Learning Functional Subspaces for Neural Network Compression

Transformers Learn the Mestre-Nagao Heuristic