Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

6,122 stories · RSS feed

arXiv Computer Vision
Aug 31

Memory-efficient GPU pipelines for real-time non-line-of-sight reconstruction

The paper presents memory‑efficient GPU pipelines that accelerate real‑time non‑line‑of‑sight (NLOS) reconstruction. By redesigning two wave‑based algorithms—f‑k migration and phasor‑fields—with fused kernels, warp‑level photon binning, batched transforms, CUDA graph replay, and selective FP16 storage, the authors achieve up to 42× speed‑ups over a reference streaming pipeline and 14× over the fastest published GPU baseline while reducing memory usage to as little as 2.5%. The work also includes an ablation study of implementation choices and introduces three denoising strategies that leverage the increased frame budget for future NLOS video processing.

By Alfonso L\'opez-Ruiz, Diego Royo
arXiv Machine Learning
Aug 31

Quantization-Triggered Backdoors in Language Models: Cross-Quantizer Transferability and the Validation--Deployment Gap

The paper shows that post‑training quantization can introduce backdoors in large language models that are not detected by source‑precision checks. By formalizing the validation‑deployment gap with Quantization Behavioral Equivalence Classes (QBECs), the authors demonstrate that models can pass full‑precision tests yet exhibit malicious behavior after INT8 or 4‑bit compression. Experiments on machine translation and political stance classification reveal significant corruption and ideological shifts, and cross‑quantizer analysis indicates that attack persistence depends on the quantization scheme and architecture rather than just bit‑width.

By Jacopo Dardini, Claudio Stanzione, Giordano Col\`o, Giuseppe Fenza
arXiv Computation and Language
Aug 31

Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

The study examines function routing for a 740‑instance healthcare API task using a 1.5B Qwen student and a 20B teacher, comparing eight knowledge‑distillation (KD) variants to supervised cross‑entropy across multiple random seeds. Results show high per‑seed variability (2.8–48.7 pp), with several KD methods exhibiting bimodal collapse—some seeds achieving low accuracy while others train normally—and distinct failure modes such as wrong‑function selection and output‑truncation. Only progressive_kd and rank_kd consistently avoid collapse, and a simple input‑enrichment trick that appeared beneficial in single‑seed tests actually harms performance when re‑tested with multiple seeds.

By Dipto Sumit, Sakib Ul Haque, Farig Sadeque
arXiv Computation and Language
Aug 31

Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

The paper investigates the intrinsic dimension (ID) of large language model (LLM) representations as an indicator of linguistic complexity. By comparing ID across model layers for coordination vs. subordination, right‑branching vs. center‑embedding, and unambiguous vs. ambiguous attachment, the authors find consistent ID differences that align with established complexity contrasts. Experiments across six LLMs, including representational similarity and layer pruning analyses, confirm that more complex phenomena produce higher ID profiles, with peaks occurring at different layers for each contrast.

By Marco Baroni, Emily Cheng, Iria de-Dios-Flores, Francesca Franzon
arXiv Computation and Language
Aug 31

PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

PRISM is a training‑free framework that efficiently selects visual instruction data for multimodal large language models by addressing the anisotropy in visual feature distributions, which causes a Global Semantic Drift. By implicitly re‑centering visual semantics, PRISM removes the influence of global background features, cutting data‑selection and model‑tuning time to 30% of conventional pipelines while improving performance across eight multimodal and three language benchmarks, achieving a 101.7% relative gain over baseline models.

By Jinhe Bi, Aniri, Zengjie Jin, Yifan Wang, Danqi Yan, Wenke Huang, Xiaowen Ma, Sikuan Yan, Artur Hecker, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, Yunpu Ma
arXiv Machine Learning
Aug 31

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

The paper introduces DAMP, a decay‑aware mixed‑precision quantization scheme for recurrent‑state representations in GDN and KDA language models. By identifying high‑risk channels through quantization‑error energy and decay persistence, DAMP stores these channels at higher precision while compressing the rest to INT8, achieving a 9.9‑bit average precision. Experiments on Qwen3.6‑35B and Kimi‑Linear‑48B show a 69.1% reduction in recurrent‑state storage, up to 2.01× faster state‑update kernels, and up to 10.9% lower full‑model TPOT while preserving accuracy close to the FP32 baseline.

By Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zixu Jiang, Yuchen Xie, Xunliang Cai, Ziqian Zeng
arXiv Computer Vision
Aug 31

Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

Token-Budget Distillation (TBD) is a parameter‑efficient fine‑tuning framework that adapts video vision‑language models to a fixed token budget. It freezes the pretrained backbone, updates only LoRA adapters, and incorporates FlashVID visual token compression. TBD uses a dual‑path teacher‑student design with full‑token supervision and compressed student optimization, enabling the student to recover full‑token semantics while remaining efficient under aggressive token reduction.

By Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, Wenhao Wang
arXiv Computer Vision
Aug 31

CF-YOLO: Context-Aware Feature Refinement for Camouflaged Industrial Micro-Defect Detection

CF-YOLO introduces a real‑time detection framework for camouflaged micro‑defects on industrial components, combining a Context‑Perception Aggregation Module (CPAM) that fuses large‑kernel macro‑texture cues with small‑kernel boundary details, and a Feature Additive Refinement Module (FARM) that globally refines fine‑grained anomaly representations. The authors also release the Copper Tube Defect Dataset (CTDD), a benchmark of 1,847 images with 4,898 annotated defect boxes. Experiments show CF‑YOLO outperforms baseline detectors such as YOLOv11 by 2.2% in mAP@50 and 3.9% in Precision while preserving real‑time speed.

By Xinda Yu, Kunxin Zheng, Chunan Yu, Qingbo Song, Hao Xiao, Ying Zang, Jie Liu
arXiv Machine Learning
Aug 31

Learning Fast Monomial Orders for Gr\"obner Basis Computations

The paper proposes treating the choice of monomial ordering for Gröbner basis computations as a reinforcement learning problem, using domain-informed reward signals that reflect actual computational cost. By training policies over the space of admissible orderings, the authors demonstrate that the learned strategies outperform traditional static heuristics such as GrevLex on benchmark problems from systems biology and computer vision. The resulting policies also resist simplification into interpretable models, suggesting that deep reinforcement learning captures complex geometric structure beyond conventional approaches.

By R. Caleb Bunch, Alperen A. Erg\"ur, Melika Golestani, Jessie Tong, Malia Walewski, Yunus E. Zeytuncu
arXiv Computation and Language
Aug 31

A Probabilistic Interpretation of KV Cache Eviction

arXiv:2608.28293v1 Announce Type: new Abstract: The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible...

By Renato Geh, Alex Chen, Daniel Israel, Aditya Grover, Guy Van den Broeck
arXiv Computer Vision
Aug 31

VersaGauss: A Versatile Framework for Generating Multiphase Dynamics with 3D Gaussians

VersaGauss is a unified framework that generates, simulates, and renders 3D dynamic scenes using 3D Gaussians, supporting multiphase interactions among materials such as fluid, rubber, sand, and snow. It takes a few input images and produces realistic, physics-driven scenes, employing a particle pruning algorithm to optimize Gaussian kernel distribution and a Coupled Multiphase Point Method (CMPM) for modeling multiphase dynamics. The framework also introduces harmonic interpolation within CMPM and a Gaussian evolution strategy to achieve realistic fluid rendering, with extensive experiments validating its performance.

By Ruijie Su, Lingxiao Yang, Xiaohua Xie, Jianhuang Lai
arXiv Computer Vision
Aug 31

Denoising-Aware Temporal Point Cloud Completion for 3D Crop Architecture Recovery and Phenotypic Trait Extraction

The paper introduces SynthCrop4D, a synthetic dataset of temporally evolving plant point clouds that includes controllable noise, occlusion, and complete geometry for benchmarking reconstruction methods. It proposes a two‑stage pipeline combining spatial denoising with an Adaptive Temporal PoinTr model to recover missing regions from self‑occlusion, achieving significant improvements in reconstruction quality on both SynthCrop4D and the real Pheno4D dataset. The completed point clouds are further used to extract phenotypic traits such as plant height, canopy width, and convex hull volume, demonstrating the pipeline’s utility for high‑throughput crop phenotyping.

By Mrudul Mittal, Soumyashree Kar
arXiv Machine Learning
Aug 31

Accelerating LLM Inference via Vector Index Based Output Embeddings

The paper proposes replacing dense output projection in large language models with an HNSW-based vector index to perform maximum inner product search over token embeddings. This approach reduces memory bandwidth usage by retrieving only a small set of high-scoring tokens and can be integrated into existing decoding pipelines via sparse logits scattering. Experiments on Gemma 3, Llama 3.2, and Qwen 3 show up to 82% speed‑up in batch‑size‑one decoding while maintaining generation quality.

By Martin Loretz, Sepp Hochreiter
arXiv Machine Learning
Aug 31

Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

The paper introduces Learned Relay Representations (Relay), a technique for Masked Diffusion Models (MDMs) that preserves and forwards internal latent information across denoising steps via a differentiable per-token channel trained with truncated backpropagation through time. Relay enables MDMs to be forward‑thinking, avoiding costly recomputation of internal representations and improving efficiency. Applied to Fast‑dLLM v2, Relay outperforms standard supervised fine‑tuning on coding tasks and reduces inference latency by up to 32%, demonstrating a clear performance‑latency advantage for diffusion language models.

By Benjamin Rozonoyer, Jacopo Minniti, Dhruvesh Patel, Neil Band, Avishek Joey Bose, Tim G. J. Rudner, Andrew McCallum
arXiv Computer Vision
Aug 31

Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models

The paper introduces Dual-Stream Semantic Guidance (DSSG), a framework for Source‑Fully‑Free Domain Adaptation of Vision‑Language Models that mitigates dual semantic drift through a caption stream and a class‑anchor stream. It adds a Dynamic Cross‑Modal Knowledge Distillation module and a Prototype Anchor Calibration extension (DSSG‑PAC) to reduce computation while maintaining performance. Experiments show DSSG outperforms state‑of‑the‑art methods and DSSG‑PAC cuts adaptation time by 18.9% with minimal loss in accuracy.

By Weiwei Xiang, Shun Peng, Guangyi Xiao, Hao Chen, Lei Yang
arXiv Computer Vision
Aug 31

Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding

The paper introduces Parallel Tube Decoding (PTD), a generative approach for spatio‑temporal video grounding that splits the task into a temporal block and simultaneous time‑conditioned spatial blocks, eliminating token‑level and trajectory‑level dependencies. PTD uses Decoupled Block Attention to allow parallel spatial generation while maintaining shared video‑query context, and incorporates localization‑aware policy optimization for temporal boundaries and spatial geometry. Experiments on VidSTG show PTD cuts tube completion latency by 79× and boosts spatial decoding throughput by 92× compared to autoregressive decoding, while improving grounding accuracy and performing well on related tasks such as temporal grounding, VideoQA, and referring video object tracking.

By Hanoona Rasheed, Haania Siddiqui, Ming-Hsuan Yang, Fahad Shahbaz Khan, Salman Khan
arXiv Machine Learning
Aug 31

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

The paper introduces VICT, a method that leverages the internal structure of verifiable tasks to perform fine‑grained credit assignment for long‑horizon LLM agents. VICT exposes executable or evidence‑backed atoms from a task’s terminal verifier and traces them back to actions via dependency‑valid proof edges, redistributing advantage only along these edges. This approach improves performance on ALFWorld and WebShop compared to outcome‑only training and matches recent fine‑grained credit methods without requiring additional critics, labels, or inference‑time verifier access.

By Pengcheng Li, Zhengyang Zhang, Dongxu Zhang, Sui Huang, Shaohua Ma
arXiv Computer Vision
Aug 31

Ex-Sim(3)-Reg: 2D-3D Correspondence Pruning via Extended Sim(3) Registration

The paper introduces Ex‑Sim(3)‑Reg, a fast and robust method for pruning 2D‑3D correspondences by reformulating the problem as an extended Sim(3) registration that explicitly accounts for depth noise. The authors provide a theoretical justification and demonstrate that their approach improves registration recall by up to 24.7% on several benchmark datasets, outperforming state‑of‑the‑art baselines. The code for the method is publicly available on GitHub.

By Pei An, Muyao Peng, Junfeng Ding, Jiaqi Yang, Liangliang Nan
arXiv Computer Vision
Aug 31

DensityKV: Density-Guided KV Cache Compression for Long Video Generation

DensityKV is a training‑free strategy for managing the historical key‑value (KV) cache in autoregressive video diffusion models. It creates a separate token‑level KV bank for each attention head and uses Soft‑Riesz density to measure and limit local redundancy among post‑RoPE keys, thereby preventing the KV archive from growing indefinitely. Experiments on three video generation backbones demonstrate that, with the same KV capacity limit, DensityKV improves long‑horizon consistency and generation stability while keeping persistent storage bounded regardless of rollout length.

By Wenqu Zhao, Xuemin Chi, Xin Zhang, Guoqing Ma, Baorun Li, Jianjie Fang, Peizhi Tang, Chen Gao, Wei Wu