The paper introduces Kernelized Parameter Sensitivity Atlases (KPSA), a method that uses a block decomposition of the empirical neural tangent kernel to estimate input‑specific parameter sensitivities efficiently. KPSA embeds parameter groups in a query space and applies a lightweight kernel to predict sensitivities for unseen inputs, achieving a high fraction of the perturbation susceptibility of full back‑propagation while dramatically reducing computation and memory usage. Experiments across vision, forecasting, and protein‑modeling tasks show that KPSA identifies sparse, depth‑spanning sensitivity circuits and improves throughput and memory efficiency in activation‑attribution problems.
arXiv:2609.27988v1 Announce Type: cross Abstract: Methods operating on Vision Transformer (ViT) feature spaces typically rely on Euclidean distance or cosine similarity. This assumes that every direc...
Methods operating on Vision Transformer (ViT) feature spaces typically rely on Euclidean distance or cosine similarity. This assumes that every direction is equally meaningful, but there is no reason...
The paper introduces MACCHIATO, a training algorithm that builds a ReLU‑MLP from partial truth‑table data while simultaneously constructing an explicit Boolean circuit over AND, OR, and XOR gates that certifies the network’s computation. The method iteratively projects residuals onto low‑dimensional Boolean classes, compiles the resulting circuit into a ReLU‑MLP, and uses logic minimization and influence‑based variable selection to achieve a six‑layer network with provable truth‑table error bounds. Experiments on synthetic random‑junta tasks show that these certified networks outperform Adam‑trained MLPs in data‑sparse or projection‑aligned regimes and complete faster than flat ESPRESSO in certain settings.
CORE-STACK+ is a new meta‑learning framework for deep stacked generalization that tackles two key problems in heterogeneous vision ensembles: prediction‑space multicollinearity and calibration collapse. It introduces a four‑step preconditioning pipeline—kernelized redundancy filtering, a lightweight differentiable meta‑feature gate, a spectrum‑adaptive ridge penalty, and a Laplace‑approximate Bayesian blender—to jointly improve conditioning and calibration. Across six vision benchmarks, CORE‑STACK+ boosts accuracy, reduces model count and inference cost, and significantly lowers expected calibration error compared to existing methods.
LILA (Latent-Informed Layer Analysis) introduces a calibration‑free method for structured pruning of large language models by scoring neuron importance using the Kolmogorov–Smirnov distance between singular value distributions of full and neuron‑ablated feed‑forward network weight matrices. The approach requires no training, calibration data, or auxiliary networks, and outperforms existing methods such as PruneNet and SliceGPT on LLaMA‑2‑7B and Phi‑2 at various sparsity levels. After a single epoch of LoRA fine‑tuning, LILA matches heavily calibrated baselines, and a Neural Tangent Kernel analysis provides theoretical support for its spectral importance criterion. Additionally, LILA can dynamically allocate sparsity budgets, achieving state‑of‑the‑art generative preservation and revealing architectural bottlenecks at higher compression.
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv:2606. 27321v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features.
arXiv:2610. 39027v1 Announce Type: cross Abstract: What computational role does overparameterisation play during learning?
arXiv:2606. 02909v1 Announce Type: cross Abstract: Gradient observations can substantially improve Gaussian process (GP) surrogates, particularly in high-dimensional settings where function evaluations are expensive.
arXiv:2609.38901v1 Announce Type: new Abstract: Representer explanations rank the training landmarks that most influence a self-supervised representation. At scale, this ranking rests on up to four s...
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
The paper investigates Active Feature Acquisition (AFA), where features are selected sequentially to maximize predictive accuracy within a budget. It introduces a theoretical certificate that decomposes the gap between a greedy Expected Information Gain minus cost (EIG-Cost) policy and the optimal hard‑budget policy into a budget‑slack term and a greedy‑suboptimality term, δ_λ. Experiments on synthetic data, standard benchmarks, and a large clinical dataset (MIMIC‑IV) show that δ_λ is close to zero in many cases, and that EIG‑Cost achieves competitive performance while using only a small fraction of the full feature cost.