arXiv:2603.14254v2 Announce Type: replace Abstract: Test-time adaptation (TTA) aims to improve model robustness under distribution shifts by adapting to unlabeled test data, but most existing methods...
Vision-language models exhibit remarkable zero-shot capabilities but suffer significant performance degradation under distribution shifts. While test-time adaptation (TTA) via Low-Rank Adaptation offers a parameter-efficient solution, we identify a fundamental bottleneck in current methods: the reliance on static rank configurations.
The paper studies how to allocate a fixed computational budget between pretraining and fine‑tuning in a two‑stage ridge regression setting. By modeling the process as a compute‑split problem and analyzing data‑dependent evaluation geometries, it derives the optimal split in terms of prediction‑relevant spectral components of the pretraining and fine‑tuning empirical covariances. The analysis uses a basis‑invariant eigenspace decomposition and perturbative control of non‑commuting dynamics to capture how pretraining directions influence downstream predictions.
arXiv:2405. 04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.
arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...
arXiv:2606. 06494v1 Announce Type: new Abstract: Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning.
The paper introduces JANUS, a post‑hoc weight rectification framework that enforces Parameter Space Orthogonality to prevent catastrophic forgetting when fine‑tuning foundation models. By projecting updates into the Jacobian Null Space and employing a Multi‑step Adaptive Rectification mechanism, JANUS dynamically verifies trust regions and adjusts step sizes. Additional techniques such as ghost projection, ghost orientation comparison, and sequence‑level SVD compression provide temporal and spatial efficiency, enabling JANUS to integrate seamlessly with various fine‑tuning methods and effectively mitigate the stability‑plasticity dilemma.
The paper surveys Continual Test-Time Adaptation (CTTA), a framework that adapts pretrained computer‑vision models to non‑stationary target distributions without source data or labeled targets, while avoiding catastrophic forgetting and error accumulation. It formally defines the CTTA problem, categorizes existing methods into optimization‑based, parameter‑efficient, and architecture‑based families, and reviews representative techniques and benchmarks across standard evaluation settings. The survey also outlines current limitations and proposes future research directions, such as adapting foundation models and black‑box systems.
arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
arXiv:2106. 06998v5 Announce Type: replace Abstract: Training convolutional neural networks at scale demands substantial memory, largely because intermediate activations must be stored for backpropagation.
arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.