Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
arXiv:2606.05122v2 Announce Type: replace Abstract: Large language models are increasingly evaluated by other models, raising a natural question: can a model predict how a judge will score its own ou...
arXiv:2608.28247v1 Announce Type: new Abstract: Change detection in Earth observation (EO) is critical for monitoring land surface transformations, yet recent research in the field is constrained by...
The paper introduces Ex‑Sim(3)‑Reg, a fast and robust method for pruning 2D‑3D correspondences by reformulating the problem as an extended Sim(3) registration that explicitly accounts for depth noise. The authors provide a theoretical justification and demonstrate that their approach improves registration recall by up to 24.7% on several benchmark datasets, outperforming state‑of‑the‑art baselines. The code for the method is publicly available on GitHub.
The study investigates whether transferring relational structure from human mental representations to deep neural networks (DNNs) can improve fine‑grained alignment between the two. Using unsupervised Gromov‑Wasserstein optimal transport, the authors show that fine‑tuning pre‑trained DNNs with Relational Knowledge Distillation (RKD) brings the networks close enough to human representations to align at the individual‑object level on a test set of concepts not seen during training. The improvement is driven mainly by a more human‑like global structure of category distances, while local nearest‑neighbor overlap remains largely unchanged.
arXiv:2608.27728v1 Announce Type: new Abstract: Diffusion models generate skillful weather ensembles but require costly iterative sampling. We introduce a supervised energy-distance distillation meth...
The paper introduces a trajectory-level speculative decoding framework for diffusion-based language models (dLLMs), addressing the limitation of existing strategies that revert to single-token generation when confidence is low. By constructing draft denoising trajectories through confidence-stratified tree exploration and verifying them with blockwise parallel evaluation and bidirectional attention masking, the method also incorporates inter-block speculation to exploit the models’ bidirectional structure. Experiments show a 30–40% reduction in denoising iterations, a token-per-step increase from 2.6 to 4.3, and a 7–14× speedup over vanilla dLLMs while maintaining accuracy within 1% on reasoning and code benchmarks.
SignRR is a new sign language production framework that combines retrieval of real sign motion segments with a learned refinement step to produce globally coherent signing sequences. It starts from a dictionary of authentic sign segments and refines them using a part-aware Residual VQ‑VAE, preserving fine hand articulation while handling temporal length differences in latent space. Experiments on PHOENIX14T and CSL‑Daily demonstrate state‑of‑the‑art back‑translation performance and competitive pose quality.
The paper develops a mathematical theory of superposition in neural networks using frame theory and compressed sensing. It shows that a sparse binary vector of active features can be encoded by an overcomplete dictionary and recovered via a ReLU operation with a suitable bias. The authors prove recovery theorems for both random-support and worst-case support settings, providing high-probability guarantees for low-coherence dictionaries and a sharp criterion for sparsity levels, with explicit results for Gaussian random matrices and equiangular tight frames.
Detecting hallucinations in Large Vision-Language Models (LVLMs) requires both accurate span localization and well-calibrated confidence scores. Fine-tuned generative VLMs excel at identifying halluci...
We show that sequence-level distillation from a capable long-context teacher model is a simple, annotation-free, and data-efficient strategy for improving argument saliency coverage in long legal opin...
The article explains how the authors eliminated the need to review every agent action by selectively directing human attention to the most critical moments. It describes the process of stopping exhaustive reviews and instead routing human oversight where it truly matters. The result is a more efficient workflow that maintains quality without sacrificing throughput.
On-policy self-distillation (OPSD) improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher that also sees a reference so...
arXiv:2608. 26418v1 Announce Type: cross Abstract: Modern AI workloads and the hardware that runs them evolve on different timescales: architectural definition precedes volume silicon by years, while target workloads shift in months.
EditaLive! is a new real‑time framework for character video editing in live streaming, built on a pretrained image animation model (Wan‑Animate) that separates appearance from motion. It uses the CharEdit‑50K dataset for reference‑frame editing and video reconstruction, and adapts the model from offline bidirectional to causal streaming generation. A self‑rollout distillation strategy compresses the model into a two‑step sampler, employing fixed RoPE, alignment forcing, and first‑frame preserved sparse attention to reduce appearance drift and achieve low‑latency inference while preserving facial expressions.
The paper introduces KDG‑SemNOMA, a framework for 6G robotic vehicle networks that combines knowledge distillation and generative models to improve semantic communication over uplink non‑orthogonal multiple access (NOMA). It employs a ConvNeXt‑based deep joint source‑channel coding architecture with an enhanced attention feature module for dynamic channel adaptation, and uses an orthogonal teacher model to guide a NOMA student model via two‑stage knowledge distillation. A channel‑conditional GAN further refines the reconstructed images, yielding higher pixel‑level accuracy and perceptual fidelity on the FFHQ‑256 dataset compared to state‑of‑the‑art methods.
SIMPLER is a pre‑fine‑tuning method that reduces inference and deployment costs for Earth Observation foundation models by pruning redundant layers. It uses layer‑wise representation similarity on unlabeled task data to identify and remove up to 79% of parameters without requiring gradients, magnitude heuristics, or hyperparameter tuning. Experiments on Prithvi‑EO‑2, TerraMind, and ImageNet‑pretrained ViT‑MAE show that SIMPLER retains 94% of baseline performance while achieving 2.1× faster training and 2.6× faster inference.
Transduced language models (TLMs) combine a pretrained source language model with a finite‑state transducer to produce a language model over target strings. The paper introduces an unbiased stochastic estimator that resamples source prefixes without replacement and reweights them, allowing accurate estimation of target prefix probabilities while reducing computation compared to threshold‑pruned beam summing. Experiments on encyclopedic text, DNA, and DNA‑to‑amino‑acid transduction show improved compute–variance trade‑offs and significant runtime reductions, and the method also lowers estimated corpus surprisal in a reading‑time analysis without altering its conclusions.
MoganColBERT-TR is a Turkish multi‑vector retrieval model that projects token embeddings from 768 to 128 dimensions and uses MaxSim late interaction for scoring. It builds on the previously trained MoganBERT‑TR encoder, adapting it to the ColBERT objective with a single‑epoch distillation phase using cross‑encoder teacher scores. Evaluated on five Turkish BEIR datasets in a zero‑shot setting, it achieves an overall score of 37.36, outperforming larger models on most datasets.
The paper introduces latent spatial memory, a 3D cache that stores scene information directly in diffusion latent space, eliminating the need for pixel-space reconstruction. It presents Mirage, a framework that lifts latent tokens into 3D using depth-guided back‑projection and queries the memory via latent‑space warping, achieving significant speed and memory gains. Experiments demonstrate up to 10.57× faster video generation, a 55× reduction in memory usage, and state‑of‑the‑art performance on WorldScore and strong reconstruction on RealEstate10K.