arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
The paper investigates a negative weight drift that arises from the interaction between standard loss functions (MSE or cross‑entropy) and positively biased activation functions. This drift, which is inherent to the optimization process rather than the data, pushes downstream weights toward negative values during early training and leads to high activation sparsity—up to nearly 90% in GPT‑nano when using ReLU. The authors also examine the impact of this sparsity, noting that model performance degrades sharply when activation sparsity exceeds about 70%, and study sparsity‑inducing activation functions that produce spikes in intermediate transformer layers.
SG-Blend introduces a per‑layer adaptive activation that interpolates between a bias‑corrected, parametric Swish variant (SSwish) and GELU, using a learnable blend coefficient, sharpness, and zero‑centering bias. The method adds only three scalars per feed‑forward block and, on BERT‑style IMDB classification, matches peak accuracy while reducing seed‑to‑seed variance by 42 %. It also achieves the lowest validation perplexity on WikiText103 and generalizes to computer vision and other domains.
Fine‑tuning reshapes internal representations of large language models, affecting attention patterns and layer‑wise activations. The study shows that components identified by EAP as important for task performance cluster in specific layers, yet these layers do not align with those undergoing the largest representational changes. Additionally, overlapping EAP components across different tasks do not guarantee cross‑task transfer and can even degrade performance when tasks differ in nature.
arXiv:2605.12484v3 Announce Type: replace Abstract: Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating model parameters force...
arXiv:2608.21664v1 Announce Type: new Abstract: Safe deployment of increasingly capable models will likely come to rely on latent-space monitoring as a complement to behavioral evaluations, especiall...
arXiv:2609.37165v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models remain brittle under visual distribution shifts, often relying on spurious correlations tied to domain-specific f...
arXiv:2609.36081v1 Announce Type: new Abstract: Representations continually change as a network learns new tasks. We ask whether early representational changes naturally form a geometric structure th...
The paper addresses shortcut learning in vision‑language‑action (VLA) models, showing that different backbones vary in their susceptibility to visual shortcuts that arise from limited demonstration diversity. It introduces a representation‑level metric called action margin to predict shortcut use without rollouts, and proposes task scrubbing—a domain‑adversarial training method—to reduce reliance on visual cues and enhance out‑of‑distribution robustness. Experiments in simulation and real‑world settings demonstrate that task scrubbing improves generalization and often eliminates visual shortcut learning.
arXiv:2606. 14259v1 Announce Type: new Abstract: Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties.
arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.
arXiv:2607. 09042v1 Announce Type: new Abstract: Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern.