Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.
Recent advances in distillation and flow-map models have enabled deterministic one- or few-step generation for high-quality data, facilitating a new branch of reward alignment approaches that directly...
arXiv:2609.36504v1 Announce Type: new Abstract: Accurate prediction of three-dimensional (3D) microstructure evolution remains computationally demanding because high-fidelity phase-field simulations...
arXiv:2609.37261v1 Announce Type: new Abstract: Softmax attention is ubiquitous in modern machine learning, but its quadratic scaling with sequence length makes it costly. To reduce this cost, attent...
LIFT is a unified image‑to‑video generation framework that adds Layout‑In‑Future control, letting users specify what should appear and where in a future view. It addresses the limitation of existing camera controls and text prompts by using the last‑frame layout as an explicit signal for the desired future scene, especially under large viewpoint changes. To handle sparse layout guidance, LIFT employs on‑policy self‑distillation to transfer knowledge from a dense‑layout teacher to a last‑frame‑layout student, and introduces the LIFT‑Vista dataset with large viewpoint changes and consistent layout annotations. Experiments demonstrate that LIFT improves video quality, future‑layout controllability, and camera controllability compared to other methods.
The study investigates which layers of feed‑forward geometry models—specifically VGGT, π³, and VGGT‑Ω—are essential for preserving camera poses and dense 3D structure. By pruning 3,018 configurations and evaluating seven metrics across indoor and outdoor datasets, the authors identify two redundancy regions (early and late) and show that combined deletions degrade performance additively, enabling more efficient pruning. They also demonstrate that CKA can serve as a cheaper proxy for interval degradation, and that closed‑form linear calibration can recover accuracy without retraining, reducing aggregator parameters by up to 44% while maintaining comparable performance.
ESMTrack is a fully end‑to‑end self‑supervised RGB‑T tracking framework that eliminates the need for costly modality‑aligned bounding boxes or offline pseudo‑label generation. It learns discriminative, temporally consistent representations using a grounding triplet loss on the initial annotated frame and a cross‑frame temporal triplet loss on unlabeled search frames, with reliable samples selected via forward‑backward consistency. A three‑branch architecture (fusion, RGB, thermal) and a modality decoupling mechanism mitigate modality dominance bias, enabling competitive state‑of‑the‑art performance, strong cross‑dataset generalization, and real‑time inference on five RGB‑T benchmarks.
The paper introduces the first systematic study of zero‑knowledge (ZK)‑friendly quantization for large language models (LLMs). It defines what makes a quantization scheme suitable for ZK proof generation and evaluates nine models, including Qwen2.5‑14B and Qwen3‑30B‑A3B, across various weight, activation, and nonlinear lookup precisions. Findings reveal that activation precision is more critical than weight precision, nonlinear lookup approximations can dominate utility loss, and that reducing bit‑width or lookup size does not always lead to proportional proving cost savings, highlighting the need for operator‑aware precision selection.
FineSID introduces a new quantization framework for semantic identifier learning in generative recommendation systems. By replacing the traditional Top‑1 hard assignment with a soft, differentiable approach, it distributes gradient updates across all codewords, leading to balanced codebook optimization and reduced identifier collisions. Experiments on public benchmarks show that FineSID improves codebook utilization and recommendation accuracy without relying on complex initialization strategies.
The paper introduces Reliable Parallel Decoding (RPD) for masked diffusion language models, addressing the unreliability of committing multiple predictions from a single forward pass. Diagnostics reveal that confidence alone is insufficient, as confident end‑sequence predictions can preempt necessary upstream computations, and downstream predictions degrade with upstream uncertainty. RPD selects candidates based on layer‑wise stability and final confidence, committing them under an entropy budget while deferring uncertain predictions, achieving superior throughput and competitive accuracy on LLaDA and Dream benchmarks.
VISTA is a method for active multimodal agents that internalizes collective visual experience via on‑policy distillation. It turns observations from multiple rollouts of the same input into shared supervision, using Collective Visual Experience Distillation (CVED) to organize observations with context and Heterogeneity‑Aware Policy Improvement (HAPI) to reinforce successful trajectories and guide learning from unsuccessful ones. The approach lets an experience‑conditioned teacher evaluate a student’s partial responses, enabling discoveries from one trajectory to inform others without altering the student’s original history, and achieves superior performance on fine‑grained perception and general reasoning tasks compared to comparable agents.
Spatial-OPSD is a label‑free self‑improvement framework for vision‑language models that leverages spatial priors such as depth, 3D relations, and camera geometry to provide dense token‑level supervision. During training, a privileged teacher uses these priors while the student learns from only the original visual‑language input, and a recursive round‑wise scheme allows repeated self‑improvement without moving the teacher. Across four VLM families, one round of Spatial‑OPSD improves the five‑benchmark average, and three rounds push a strong spatially specialized model to the open‑source frontier, achieving the highest average among open models and best results on three of five spatial reasoning benchmarks.
arXiv:2609.36760v1 Announce Type: new Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memor...
arXiv:2609.36601v1 Announce Type: new Abstract: On-policy distillation (OPD) reduces train-test state mismatch by training a student on its own generated trajectories, but weak students may visit tea...
AnyAct introduces a universal action layer that consolidates diverse tool capabilities into a self‑evolving action space for AI agents operating in open‑world environments. It tackles the scale dilemma, tool non‑stationarity, and heterogeneous feedback by using hierarchical progressive retrieval and test‑time reliability evolution, while a heterogeneous observation grounding module unifies multi‑modal feedback. Evaluations on LiveMCPBench and the newly created OSMCP benchmark show state‑of‑the‑art performance, with significant gains in task success rate and reduced execution steps, especially for models with limited native capabilities.
arXiv:2609.35881v1 Announce Type: cross Abstract: Can a genome model retain a detectable record of the synthetic sequences used to train it? We study watermark inheritance through distillation with G...
Reference-Guided Machine Unlearning (ReGUn) is a vision unlearning framework that prioritizes distributional indistinguishability over degradation-based heuristics. It uses disjoint held-out data to create a class-conditioned reference distribution for distillation, guiding forget samples toward non-member behavior without explicitly degrading predictions. Experiments across various architectures and datasets show that ReGUn achieves a competitive forgetting–utility trade-off and closely matches retrain-like membership inference behavior.
The paper introduces Federation of Experts (FoE), a new architecture that reorganizes the mixture-of-experts (MoE) block in transformer layers into multiple MoE clusters. Each cluster handles a single KV head, and expert parallelism is applied within clusters while a sum operation synchronizes post‑attention residuals across clusters. FoE eliminates all‑to‑all communication on a single GPU and limits it to intra‑node communication in multi‑node setups, leading to significant reductions in inference latency and throughput improvements on LongBench.
The paper introduces On-Policy Attention Self-Distillation (OPASD), a method that augments token-level supervision with solution-conditioned attention distillation for reasoning models. OPASD projects a privileged teacher’s attention onto student-visible positions, renormalizes the distribution, and aligns it with the student. Experiments on three model sizes and four math benchmarks show that OPASD improves accuracy by 4.98–8.40 percentage points, reduces generated tokens by 73.9%, cuts compute by 72.6%, and trains 1.53× faster compared to token-only distillation.
arXiv:2609.35750v2 Announce Type: replace-cross Abstract: Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been...
The paper introduces DAN, a training‑free inference‑time framework that improves affective reasoning in multimodal large language models. It combines a Hierarchical Emotional Reasoning Chain (HERC) to better capture fine‑grained visual cues and a Contrastive Discriminative Visual Pruning (CDVP) module to isolate discriminative tokens for semantically similar emotions. Experiments show significant gains, notably a +10.47% improvement on the WebEmo25 benchmark with Qwen3‑VL‑8B‑Instruct.