arXiv Machine Learning
Jul 2
FastUMAP: Scalable Dimensionality Reduction via Bipartite Landmark Sampling
arXiv:2605. 11428v2 Announce Type: replace Abstract: Exploratory analysis of high-dimensional data rarely stops at a single embedding.
Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs
The paper studies Online Kernel Supervised Principal Component Analysis (OKSPCA), which uses random features and an Adam-style orthonormal basis update to optimize a supervised spectral objective. It shows that accurate optimization of this objective does not guarantee accurate population subspace recovery or improved predictive performance, and it provides theoretical results on consistency, concentration, and perturbation of the estimator. Empirical experiments on six benchmarks reveal that replacing the tracker with the exact empirical target does not significantly change regression deficits, while classification-rank models capture most of the terminal objective energy but can exhibit substantial geometric deviation; sample-size studies further separate empirical accuracy from population recovery. The diagnostics also compare computational trade-offs, indicating that exact on-request computation can be faster in classification settings, whereas Adam saves time relative to full thin‑SVD in some dense regression requests, despite persistent geometric error.
Related stories
arXiv AI
Jun 15
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
arXiv AI
Sep 18
Green-ELM: Efficient Analytic Learning via High-Dimensional Random Projections
arXiv:2604. 15613v4 Announce Type: replace-cross Abstract: We present Green-ELM, a non-iterative neural architecture that replaces gradient-based optimization of the output layer with a closed-form analytic solution over a fixed, high-dimensional random feature representation.
arXiv AI
Sep 10
Revisiting Thinning Methods for Kernel Learning Problems
The paper introduces Backward Kernel Herding, an algorithm that iteratively removes data points to create representative subsets for kernel learning, achieving performance comparable to state‑of‑the‑art methods while speeding up subsampling when the reduced size is less than half the original dataset. It also proposes Flexible Kernel Thinning, an extension that allows construction of subsets of any size, not just successive halvings, and demonstrates that this method often yields the best predictive performance. Experiments on Gaussian Processes and Kernel Support Vector Machines show that Backward Kernel Herding excels in training‑time efficiency, while Flexible Kernel Thinning offers superior predictive accuracy and competitive memory usage, emphasizing the need to choose a reduction strategy based on the desired trade‑off between performance, cost, and memory.
arXiv Machine Learning
6d ago
Stacked SVD or SVD stacked? A Random Matrix Theory perspective on data integration
The paper compares two popular data‑integration techniques—Stack‑SVD, which concatenates datasets before performing singular value decomposition, and SVD‑Stack, which first decomposes each dataset separately and then aggregates the leading singular vectors. By deriving exact asymptotic performance expressions and phase transitions in a proportional regime, the authors show that neither method uniformly dominates the other when unweighted, but optimally weighted Stack‑SVD outperforms optimally weighted SVD‑Stack when the low‑rank signal is fully shared. They also demonstrate that SVD‑Stack can excel with partially shared components and provide practical algorithms for estimating optimal weights, supported by simulations and genomic experiments.
arXiv AI
Jun 16
Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning
arXiv:2510. 16882v4 Announce Type: replace-cross Abstract: Supervised fine-tuning (SFT) is a commonly used technique to adapt large language models (LLMs) to downstream tasks.
arXiv Machine Learning
Sep 15
Rotation-Based Subspace Tracking for Robust Kernel PCA on Streaming Data
arXiv:2609.15488v1 Announce Type: new Abstract: Machine learning models process large amounts of data, and Principal Component Analysis (PCA) is a widely used technique to reduce the dimensionality o...
arXiv Machine Learning
1d ago
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression
arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...
arXiv Machine Learning
Jun 17
Reducing Learner Redundancy in Boosting via Residual Orthogonalization
arXiv:2606. 17567v1 Announce Type: new Abstract: While sequential residual fitting is the bedrock of standard boosting frameworks, it inherently breeds learner redundancy by repeatedly revisiting correlated error components.
arXiv AI
Aug 18
Validation-Frontier Representation Selection under Constrained Observation
arXiv:2608. 15095v1 Announce Type: new Abstract: AI systems deployed outside clean benchmark settings often rely on observations that are incomplete, unstable, costly, or degraded by monitoring failures.
arXiv AI
Jul 7
Panorama: Fast-Track Nearest Neighbors
arXiv:2510. 00566v4 Announce Type: replace-cross Abstract: Approximate Nearest-Neighbor Search (ANNS) pipelines for high-dimensional neural embeddings spend the bulk of their query time in candidate verification, making it the primary bottleneck in the search process.
arXiv Machine Learning
Jul 21
De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction
arXiv:2607. 16638v1 Announce Type: cross Abstract: Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues.