arXiv Machine Learning June 4, 2026 By Ambroise Odonnat, Laetitia Chapel, Romain Tavenard, Ievgen Redko

Vision Transformer Finetuning Benefits from Non-Smooth Components

Read the original on arXiv Machine Learning →

arXiv:2602. 06883v3 Announce Type: replace Abstract: The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 9

A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers

arXiv:2606. 07593v1 Announce Type: cross Abstract: The widespread use of image classification models in high-risk, real-world situations necessitates making these models robust to slight disturbances or perturbations, such as blurring or sharpening, in the input images.

By Hannah Gao (Massachusetts Institute of Technology), Isha Agarwal (Massachusetts Institute of Technology), Dylan Hadfield-Menell (Massachusetts Institute of Technology), Rachel Ma (Massachusetts Institute of Technology)

llms computer-vision fine-tuning multimodal safety

arXiv AI

Sep 15

Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

arXiv:2609.15229v1 Announce Type: cross Abstract: We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains...

By Qingtao Xia, Jiahua Bao, Siyao Cheng, Jie Liu

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 10

Stability of Transformers under Layer Normalization

arXiv:2510. 09904v2 Announce Type: replace-cross Abstract: Despite their widespread use, training deep Transformers can be unstable.

By Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Krishna Kumar, Markos A. Katsoulakis

llms

arXiv Machine Learning

Jul 14

AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

arXiv:2607. 10593v1 Announce Type: new Abstract: Normalization is a critical component for stabilizing Transformer training, yet the choice between static strategies such as Layer Normalization (LN) and adaptive alternatives remains largely task-dependent.

By Piyush Kaushik Bhattacharyya, Divyanshu Rai, Swastik Singh, Kumar Aakash, Ayush Ranjan, Krutika Verma

llms nlp benchmarks

arXiv Machine Learning

4d ago

Procedural Core: A Compact Recurrent Initialization for Vision Transformers

arXiv:2609.37631v1 Announce Type: new Abstract: Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work sh...

By Zachary Shinnick, Christian Intern\`o, Hemanth Saratchandran, Anton van den Hengel, Damien Teney

llms computer-vision

arXiv AI

Aug 19

Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization

The paper investigates why adaptive optimizers like Adam outperform SGD when fine‑tuning Transformers. It introduces gradient heterogeneity—the variation in gradient norms across parameter blocks—and shows, both theoretically and experimentally, that this heterogeneity, together with Hessian heterogeneity, hampers SGD convergence while sign‑based methods such as SignSGD are less affected. The study links the source of gradient heterogeneity to layer‑normalization placement, finding that Post‑LN architectures exhibit the strongest effect, and uses SignSGD as a tractable proxy to analyze Adam‑like behavior and learning‑rate scaling.

By Akiyoshi Tomihari, Issei Sato

llms nlp fine-tuning

One email a morning, machine-written

Related stories

A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers

Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

Stability of Transformers under Layer Normalization

AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

Procedural Core: A Compact Recurrent Initialization for Vision Transformers

Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization