← Back to all news
arXiv Machine Learning By Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu

Rethinking Sequence Compression with Test-time Regression

Read the original on arXiv Machine Learning →

The paper introduces TTR‑Compressor, a new method for sequence compression that treats token aggregation as a test‑time regression problem. Unlike traditional pooling or learned projections, the regression coefficients are learned end‑to‑end and computed in closed form during inference. Experiments show that TTR‑Compressor outperforms existing compressors for block‑wise sparse attention and visual token merging, achieving better performance with fewer tokens.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 9

V-CoLA: Vision Token Compression with Linear Attention

The paper introduces V-CoLA, a training‑free token compression framework tailored for vision-language models that use linear attention. V-CoLA employs a uniqueness‑aware importance criterion to select critical vision tokens and an adaptive token merging strategy to compress them, all optimized to fit the chunk‑wise parallelism of linear attention. Experiments show that V‑CoLA retains 99.5% of the original performance with only half the vision tokens and achieves significant speedups during prefill.

By Hao Jiang, Yiru Mao, Tianpeng Bu, Hao Zhou, Hongtao Duan, Wang Jing, Bowen Xu, Xin Chen, Lulu Hu, Bin Yang, Yongliang Tao, Minying Zhang
llmsmultimodalbenchmarks
More like this →
arXiv AI
Oct 1

Switching Linear Attention

Switching Linear Attention (SwiLA) is a new sequence layer that improves upon standard softmax attention by maintaining a fixed-size recurrent state while enhancing representational capacity. It derives its recurrence from a test-time regression framework, using online expectation-maximization in a mixture of linear regressions model. In various benchmarks—including associative recall, in-context language learning, and language modeling—SwiLA achieves strong performance, narrowing the gap to softmax attention and even surpassing it in some settings.

By Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet, E. Kelly Buchanan, Emily B. Fox, Scott W. Linderman
efficiencybenchmarks
More like this →
arXiv Computation and Language
Aug 28

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.

By Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim
llmsdiffusionmultimodalbenchmarks
More like this →