arXiv:2605. 04913v4 Announce Type: replace-cross Abstract: LLM post-training typically propagates task gradients through the full depth of the model.
arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informa...
arXiv:2608.31096v1 Announce Type: cross Abstract: Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while...
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
arXiv:2607.01630v2 Announce Type: replace Abstract: Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or subnetworks, yet our...
Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or subnetworks, yet our analyses suggest that classification supervision alone does not sufficiently preserve task-agnostic shared backbone representations over long incremental sequences. We identify two intertwined challenges: cross-task confusion from sequential training on predominantly current-task data, which biases decision boundaries toward recent tasks; and under-optimized shared representations in the backbone that cap long-term discriminability as tasks accumulate.
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
arXiv:2607. 21291v1 Announce Type: cross Abstract: Large language models (LLMs) achieve strong generation and reasoning performance, but the Transformer architecture incurs high inference cost.
arXiv:2507.00754v3 Announce Type: replace Abstract: The integration of Large Language Model (LLMs) blocks with Vision Transformers (ViTs) holds immense promise for vision-only tasks by leveraging the...
arXiv:2605.12491v2 Announce Type: replace Abstract: Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design im...
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
The paper introduces Activation Boundary Matching for Low‑Rank Adaptation (ABM‑LoRA), a task‑informed initialization strategy that uses the signs of layer‑wise pre‑activations from a brief probe adapter as targets for a fresh adapter. By training with a margin‑based hinge objective on these activation boundaries, ABM‑LoRA captures useful adaptation directions that standard LoRA initializers miss, while requiring only a few forward passes. Experiments show that ABM‑LoRA outperforms or matches existing LoRA, SVD, and gradient‑based initializers across multiple models and benchmarks, including T5‑base/GLUE, ConvNeXt‑T, Swin‑T, Qwen2.5‑1.5B, and LLaMA2‑7B.