Pixels to Tokens introduces TEAL, a token‑space efficient active learning framework that operates at the pixel level for semantic segmentation using a frozen Vision Transformer backbone. TEAL selects diverse candidate pixels via a two‑level MaxHerding strategy on multi‑layer token descriptors and refines uncertainty on a finer decoder grid, avoiding interpolation artifacts. Across multiple datasets, TEAL outperforms prior baselines under extreme label scarcity, achieving up to +24.36 mIoU with a single labeled pixel per image and reaching 97% of fully supervised performance with only ten pixels.
arXiv:2606.16996v2 Announce Type: replace-cross Abstract: Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabu...
The paper proposes a lightweight active‑vision pipeline that mimics biological foveation to perform semantic segmentation more efficiently. By selecting salient fixations, observing high‑resolution foveal patches, and using low‑resolution context, the method achieves 95.9% of baseline Top‑1 accuracy with only 4.7% of the computational cost, and recovers 90.6% of baseline object recall using 58.6% of the computation. The results demonstrate that sparse, selectively allocated observations can yield substantial semantic understanding, suggesting active vision as a viable alternative to uniform dense processing.
arXiv:2606. 16996v1 Announce Type: cross Abstract: Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes.
The paper proposes a lightweight active‑vision pipeline that mimics biological foveation to perform dense semantic segmentation more efficiently. By selecting salient fixations, it uses high‑resolution foveal views, low‑resolution context, and adaptive computation to accumulate semantic information. On ADE20K‑Object, a single foveated observation attains 95.9% of baseline Top‑1 accuracy with only 4.7% of the computational cost, and semantic accumulation recovers 90.6% of baseline recall using 58.6% of the computation.
The paper studies Graph-Guided Token Merging (G2TM), a module that reduces token count in Vision Transformers. It evaluates G2TM across multiple segmentation frameworks and decoder types, finding that its performance gains are tied to the encoder rather than the decoder. The authors report consistent reductions in GFLOPs (22‑47%) and throughput improvements (up to 74%) on ADE20K, with optimal hyperparameters depending mainly on backbone pre‑training and target dataset.
arXiv:2608. 07088v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive.
PACE introduces a training‑free Condense‑and‑Extract framework that speeds up Vision‑Language Model inference by first adaptively downsampling visual inputs before encoding and then selectively retaining essential tokens during decoding. The Adaptive Pixel Compressor (APC) reduces encoder workload while preserving global context, and the Dynamic Dual‑Attention Extractor (DDAE) keeps task‑critical details by fusing visual and language signals. Applied to Qwen2.5‑VL‑7B, PACE maintains 93.8% of performance using only 10% of visual tokens, achieving a 3.1× speedup in time to first token.
arXiv:2608. 04515v1 Announce Type: cross Abstract: Slice-based MLLMs leverage mature 2D encoders by representing 3D volumes as sequences of 2D slices.
arXiv:2609.16656v1 Announce Type: new Abstract: State space models (SSMs), particularly Mamba, have emerged as efficient alternatives to attention-based architectures and have been extended to vision...
arXiv:2512. 08240v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) rely on hundreds of visual tokens, leading to high computational and memory costs.
arXiv:2609.39924v1 Announce Type: cross Abstract: Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, m...
arXiv:2610.17412v1 Announce Type: cross Abstract: Vision Transformers incur high computational costs when processing long visual token sequences. Existing reduction methods typically rank tokens inde...