arXiv AI July 24, 2026 By Yijun Liang, Yunjie Tian, Yijiang Li, Yuqi Jia, Furong Huang, Tianyi Zhou, Di Fu

Visual Contrastive Self-Distillation

arXiv:2607. 21556v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

2d ago

Self-Supervised Visual On-Policy Distillation

arXiv:2608. 14144v1 Announce Type: cross Abstract: Visual on-policy distillation relies heavily on an informative teacher-student asymmetry, through either a larger, stronger teacher or privileged supervision, such as reference answers or ground-truth regions of interest.

By Yijiang Li, Yijun Liang, Yunjie Tian, Bingyang Wang, Ke Zhang, Zhenfei Yin, Di Fu, Philip Torr, Nuno Vasconcelos

llms efficiency benchmarks

arXiv Machine Learning

Jun 5

ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

arXiv:2606. 05718v1 Announce Type: cross Abstract: On-policy distillation (OPD) improves reasoning by training a student on trajectories sampled from its own policy under supervision from a teacher.

By Kanghui Tian, Siyuan Liu, Ziang Yan, Sheng Xia, Shuai Dong, Yi Wang

efficiency multimodal benchmarks

Hugging Face Trending Papers

Jun 17

Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning

Unified multimodal models (UMMs) interleave generated ''visual thoughts'' (VTs) with text reasoning to improve spatial tasks. This incurs roughly an order-of-magnitude inference cost from multi-step diffusion.

diffusion efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.

By Leyan Xue, Feng Xiong, Mingjun Ma, Changqing Zhang

llms efficiency multimodal benchmarks

arXiv AI

Jun 3

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

arXiv:2605. 18740v4 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image.

By Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, Yaojie Lu

llms agents efficiency multimodal benchmarks

arXiv Machine Learning

Jun 11

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

arXiv:2606. 11709v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution.

By Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, Lijie Wen

reinforcement-learning efficiency

Related stories

Self-Supervised Visual On-Policy Distillation

ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation