AdaKerNet is a task‑adaptive neural kernel decoder that operates on frozen multimodal representations from large foundation models, without requiring access to the models’ parameters. It learns Lipschitz‑controlled multimodal features, a reference kernel providing a soft structural prior, and a lightweight nonlinear predictor that deforms this structure. Experiments on four multimodal large language models and diverse input modalities show consistent improvements over baseline decoders, achieving up to 41% error reduction in scarce‑label settings.
By Konstantinos D. Polyzos, Eleni Oikonomou, Tara Javidi
arXiv:2608.24482v1 Announce Type: cross
Abstract: Mechanistic Localization bridges mechanistic interpretability and post-training optimization by isolating critical parameters via interpretative appr...
By Hang Chen, Jiaying Zhu, Wenya Wang
arXiv:2510. 00399v2 Announce Type: replace Abstract: The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks.
By Hongkang Li, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, Meng Wang
Mechanistic Localization bridges mechanistic interpretability and post-training optimization by isolating critical parameters via interpretative approaches and then guiding parameter-efficient Supervi...
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.
By Weinuo Ou
The paper introduces an individualized sparse regression framework for matrix‑valued covariates, where each observation has its own relevant rows while regression effects are shared across the population. It proposes a diagonalized attention mechanism that uses query–key scores to localize sample‑specific signal rows and a value matrix for downstream regression, achieving a parameter dimension independent of sample size. The authors provide existence theorems guaranteeing recovery of latent rows under score‑separation and concentration conditions, and demonstrate strong prediction, localization, and classification performance in simulations and real sentiment analysis.
By Borui Peng, Liwei Lin, Feifei Wang, Long Feng
arXiv:2603.10305v4 Announce Type: replace
Abstract: Machine learning models can represent climate processes that are nonlocal in horizontal space, height, and time, often by combining information acr...
By Savannah L. Ferretti, Jerry Lin, Sara Shamekh, Jane W. Baldwin, Michael S. Pritchard, Tom Beucler
The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.
By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu
The paper investigates diffusion models trained in a lazy high‑dimensional regime, extending benign overfitting theory to generative settings. By analyzing denoising score matching in a vector‑valued RKHS with an inner‑product kernel, the authors derive exact risk trajectories under gradient flow when the number of samples scales proportionally with dimensionality. These trajectories reveal three distinct phases—spectral generalization, noise‑dominated interpolation, and empirical Bayes memorization—whose interplay shapes the distribution of generated samples.
By Hugo Latourelle-Vigeant, Sinho Chewi, Aram-Alexandre Pooladian, John Sous, Theodor Misiakiewicz
The paper introduces Flow-based Spectral Kernel Attention (FSKA), a framework that learns nonstationary attention kernels using spectral density learning and normalizing flows. FSKA replaces traditional query–key projections with a simplified design that retains fixed orthogonal value projections, leading to a more parameter‑efficient architecture. Experiments on classification tasks demonstrate competitive or improved performance while reducing trainable projection parameters by roughly 40%–47% and preserving linear‑attention memory scaling.
By Zi Yang, Ying Li, Alejandro Lancho, Dar\'{i}o C. Larese, Pablo M. Olmos, Michael Minyi Zhang
arXiv:2606. 00557v1 Announce Type: new Abstract: To understand how a neural network (NN) functions and makes predictions, it has become increasingly clear that analyzing only the input domain is insufficient -- one must also examine its internal inference mechanisms to capture the complete picture.
By Ping Xiong, Thomas Schnake, Gr\'egoire Montavon, Klaus-Robert M\"uller, Shinichi Nakajima
arXiv:2506. 08297v2 Announce Type: replace-cross Abstract: Attention is the critical component of a transformer.
By Nhat Thanh Tran, Fanghui Xue, Shuai Zhang, Jiancheng Lyu, Yunling Zheng, Yingyong Qi, Jack Xin