arXiv Computer Vision October 23, 2026 By Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, Qingming Huang

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Read the original on arXiv Computer Vision →

The paper introduces CODI, a method that enhances self‑supervised Vision Transformers by mixing a small portion of the teacher’s class‑token distribution into each teacher patch‑token target. This cross distillation approach gives every masked patch an attention‑independent semantic correction without altering the architecture. Experiments demonstrate that CODI improves performance on both dense prediction and image‑level tasks under aligned pretraining settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Hugging Face Trending Papers

3d ago

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Self-supervised Vision Transformers learn strong image-level representations, but their patch-level representations can be unreliable for dense prediction. We study this mismatch and find that the cla...

llms efficiency benchmarks

arXiv Computer Vision

Sep 1

Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs

arXiv:2507.00754v3 Announce Type: replace Abstract: The integration of Large Language Model (LLMs) blocks with Vision Transformers (ViTs) holds immense promise for vision-only tasks by leveraging the...

By Selim Kuzucu, Muhammad Ferjad Naeem, Anna Kukleva, Federico Tombari, Bernt Schiele

llms fine-tuning

arXiv Computer Vision

Oct 2

Subtoken Vision Transformer for Fine-grained Recognition

arXiv:2607.09086v2 Announce Type: replace Abstract: We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transfo...

By Jie Zhu, Ivy Zhang, Minchul Kim, Xiaoming Liu

llms computer-vision nlp

Hugging Face Trending Papers

Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

llms efficiency benchmarks safety

arXiv AI

Oct 15

Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

arXiv:2610.16817v1 Announce Type: cross Abstract: Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large languag...

By Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi, Karl Henrik Johansson, Kaitlin Maile, Johannes von Oswald, Stefan Bauer

llms

arXiv AI

Jun 2

On the Limits of Token Reduction for Efficient Unified Vision Language Training

arXiv:2606. 01503v1 Announce Type: cross Abstract: Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective.

By Siyi Chen, Weiming Zhuang, Jingtao Li, Lingjuan Lv

llms multimodal

One email a morning, machine-written

Related stories

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs

Subtoken Vision Transformer for Fine-grained Recognition

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

On the Limits of Token Reduction for Efficient Unified Vision Language Training