← Back to all news
arXiv Computer Vision By Madhav Iyengar, Andreas Geiger, Hilde K\"uhne

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

Read the original on arXiv Computer Vision →

The paper introduces a method to retrofit depth recurrence into pretrained vision‑language models, reducing the language backbone’s parameter count by pruning layers and looping over the remaining ones with linear adapters. A three‑stage recovery training—language‑only pretraining, vision‑language projector alignment, and full fine‑tuning—helps recover performance, with distillation from the non‑recurrent model further improving results. Experiments on TinyLLaVA‑1.4B show the retrofitted recurrent model outperforms the unpruned baseline while saving 23.3% of language parameters and performs better on reasoning‑oriented benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
Aug 26

VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference

VisCache introduces a two-stage, plug‑and‑play framework for pruning visual key‑value caches in Vision Large Language Models without retraining. The first stage filters out temporally redundant keyframes, while the second stage, PruneKV, applies a parabolic layer‑wise budget and asymmetric update to selectively prune keys and fuse values, preserving essential context. Experiments show up to 2.35× speedup and significant memory savings with only 19–28% of the original cache retained, outperforming existing baselines.

By Lyuke Wang, Zhuo Li, Guangxu Zhu
llmsefficiencymultimodal
More like this →
arXiv Computer Vision
2d ago

AdaLoop: Thinking with the Vision Encoder

AdaLoop introduces a question‑conditioned visual recurrence framework for vision‑language models, allowing the vision encoder to adapt its computation to each question while keeping pretrained weights frozen. The method reuses encoder transformations across multiple iterations, improving performance over standard fine‑tuning and single‑pass baselines. Experiments on the VisRO dataset demonstrate gains across three VLM families on both in‑distribution and out‑of‑distribution examples.

By Xinran Nicole Han, Yiyang Lu, Katelyn Gan, Todd Zickler
llmsfine-tuningmultimodal
More like this →
arXiv AI
Aug 28

PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

PACE introduces a training‑free Condense‑and‑Extract framework that speeds up Vision‑Language Model inference by first adaptively downsampling visual inputs before encoding and then selectively retaining essential tokens during decoding. The Adaptive Pixel Compressor (APC) reduces encoder workload while preserving global context, and the Dynamic Dual‑Attention Extractor (DDAE) keeps task‑critical details by fusing visual and language signals. Applied to Qwen2.5‑VL‑7B, PACE maintains 93.8% of performance using only 10% of visual tokens, achieving a 3.1× speedup in time to first token.

By Junjie Liu, Shengyuan Ye, Xu Chen
llmsefficiencymultimodal
More like this →