arXiv AI
Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computer Vision
Sep 1
Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs
arXiv:2507.00754v3 Announce Type: replace Abstract: The integration of Large Language Model (LLMs) blocks with Vision Transformers (ViTs) holds immense promise for vision-only tasks by leveraging the...
arXiv Computer Vision
Sep 24
Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning
arXiv:2609.28187v1 Announce Type: new Abstract: Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, y...
arXiv Computer Vision
Oct 2
Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores
arXiv:2605.12491v2 Announce Type: replace Abstract: Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design im...
arXiv AI
Jun 15
Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
arXiv Computer Vision
Sep 7
DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models
DART is a new RGB‑D pretraining method for surgical vision foundation models that incorporates pseudo‑labeled depth maps as a pixel‑space reconstruction target during training. By adding a depth reconstruction head to DINOv2’s masked iBOT framework, DART improves representation quality without affecting downstream RGB‑only fine‑tuning or inference. Across eight surgical benchmarks—including segmentation, depth estimation, and image‑level recognition—DART outperforms both natural‑image and in‑domain baselines, demonstrating that geometric pseudo‑labels can strengthen foundation model pretraining without extra labels or inference cost.
arXiv Computer Vision
2d ago
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
The paper introduces CODI, a method that enhances self‑supervised Vision Transformers by mixing a small portion of the teacher’s class‑token distribution into each teacher patch‑token target. This cross distillation approach gives every masked patch an attention‑independent semantic correction without altering the architecture. Experiments demonstrate that CODI improves performance on both dense prediction and image‑level tasks under aligned pretraining settings.