Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.
arXiv:2608. 26004v1 Announce Type: cross Abstract: Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions.
BRIDLE is a self‑supervised encoder pretraining framework that extends bidirectional training to audio, image, and video by incorporating residual quantization (RQ) with multiple hierarchical codebooks. This approach allows fine‑grained discretization of latent representations and interleaves training between the encoder and tokenizer. Experiments show that BRIDLE achieves state‑of‑the‑art results on audio classification benchmarks and competitive performance on image and video classification tasks, outperforming traditional vector‑quantization methods.
The paper introduces a lightweight multimodal vision‑language framework based on TinyCLIP for fine‑grained classification of early‑stage apple fruitlet anatomy (calyx, fruitlet body, peduncle) in orchard images. Using a dataset of 600 high‑resolution RGB images, the model employs domain‑specific language prompts and a sliding‑window inference strategy to produce interpretable heatmaps for whole‑image localization. Achieving macro‑F1 of 0.93 on an NVIDIA T4 GPU and maintaining accuracy after INT8 quantization, the system is optimized for edge deployment on NVIDIA Jetson hardware with model sizes around 127‑137 MB and millisecond‑level inference.
MLLMCLIP introduces a heterogeneous distillation framework that transfers multimodal knowledge from a generative Multimodal Large Language Model (MLLM) teacher directly into a discriminative CLIP student, eliminating the need for synthetic hard negatives. The method uses an attention-based per-layer token selection and a CKA-based distillation loss to bridge architectural differences between the two models. As a result, MLLMCLIP achieves state‑of‑the‑art compositional accuracy and improves zero‑shot classification and image‑text retrieval performance.
The paper introduces Adaptive Log‑Space (AL) quantization, a block‑wise representation that adapts the non‑zero range per block and preserves the exact‑zero invariant for non‑negative optimizer states. AL8 and AL16 are combined with signed‑momentum encodings and state‑specific precision choices, rather than a single policy for all states. Experiments on TinyLlama‑1.1B and GPT‑2 show that AL‑based quantization can match or exceed full‑precision performance while dramatically reducing optimizer‑state storage.
The paper investigates how parameter‑efficient fine‑tuning affects a language model’s ability to learn new facts while retaining existing knowledge. Using a controlled OpenStreetMap benchmark, the authors compare full fine‑tuning with quantized low‑rank adaptation (QLoRA) at various ranks, finding a clear trade‑off: lower ranks preserve out‑of‑domain performance but acquire fewer facts, whereas higher ranks improve factual generalization at the cost of unrelated benchmark performance. Diagnostics in distribution, weight space, and spectral analyses confirm this acquisition‑retention frontier, and a separate math adaptation experiment shows a weaker effect, indicating the phenomenon is strongest when installing new factual associations.
BanglaMamba explores Mamba-based State Space Models (SSMs) as a computationally efficient alternative for Bangla fake news detection. Compared to BanglaBERT and a custom BERT trained from scratch, BanglaMamba achieves a Macro‑F1 score of 0.9029, close to the 0.9057 of the custom BERT, while delivering 2.2× higher inference throughput and 49% lower peak GPU memory usage. Cross‑dataset evaluation shows BanglaBERT generalizes better, underscoring the value of large‑scale pretraining.
MMEmb-R1 is a multimodal embedding framework that enhances reasoning by treating it as a latent variable and selecting beneficial reasoning paths through pair-aware selection and counterfactual intervention. It uses reinforcement learning to invoke reasoning only when necessary, reducing unnecessary computation and latency. On the MMEB-V2 benchmark, MMEmb-R1 achieves a state‑of‑the‑art score of 71.2 with just 4 B parameters.
Emergent Abilities in Large Language Models: A Survey reviews how scaling LLMs leads to previously unseen capabilities such as advanced reasoning, in-context learning, coding, and problem-solving. The paper critically examines definitions, inconsistencies, and the conditions that foster these abilities, including scaling laws, task complexity, pre‑training loss, quantization, and prompting strategies. It also discusses the extension to Large Reasoning Models and highlights safety concerns like deception, manipulation, and reward hacking, calling for improved evaluation and governance.
TOPAS is a Task‑Oriented Prefix‑Aware Scheduler designed for multi‑agent large language model serving. It jointly decides which agent prefixes to retain in a shared key‑value cache and which requests to schedule, balancing the reduction of each task’s longest remaining service path against the benefit of downstream prefix reuse while accounting for movement and preemption costs. Experiments on synthetic DAGs and MetaGPT software‑development workflows show that TOPAS can reduce mean and p99 job completion times by up to 39.8%/49.4% and 22.0%/26.6% respectively compared to the best baselines.
SAMpLE is an open‑source SystemC‑AMS framework that treats machine learning models as first‑class Timed Dataflow components via a standardized plug‑and‑play interface. It offers a native C++ backend for online training of lightweight models and an offline backend that runs externally developed models without re‑implementation. By using ONNX as a model exchange format, SAMpLE enables the integration and evaluation of diverse ML solutions within a single, reproducible simulation workflow.
The paper introduces Interaction‑Aligned Pruning (IAprune), a training‑free method for visual token pruning in embodied manipulation tasks. IAprune jointly decides per‑frame budget and token selection, using semantic‑motion spatial agreement to choose between conservative and aggressive coverage, and applies geometric residual correction to focus on under‑represented boundaries. Experiments on four policies, three simulation benchmarks, and a real‑robot platform show that IAprune matches unpruned performance on LIBERO while achieving up to 1.54× speed‑up and 1.48× acceleration on a real robot.
The paper shows that only a small subset of attention heads in vision-language models is responsible for selecting critical visual tokens. By pruning tokens based on similarity before LLM reasoning and then applying head‑aware pruning during reasoning, the proposed ProViP framework achieves high task performance with significant speedups. Experiments on LLaVA‑1.5‑7B demonstrate 95.9% performance retention and a 1.62× inference speedup at an 88.9% pruning ratio.
The paper introduces FAMPWQ, a Fisher information-based Adaptive Mixed Precision Weight Quantization method designed to improve Large Language Model inference on commodity GPUs. It uses a Fisher information metric to assess layer-wise sensitivity and a reinforcement learning-based bit-width allocator to adaptively assign precision per layer. Experiments across seven models and five benchmarks show significant gains in perplexity, accuracy, and LLM-as-a-judge performance compared to seven baseline approaches.
FLEET is a token‑based feature extractor that processes event camera data directly, using random Fourier features and cross‑attention to compress variable‑length event streams into fixed‑size latent representations. By decoupling inference cost from sensor resolution, it avoids the high compute and temporal blurring associated with CNN‑based grid aggregation. Experiments on a new high‑throughput benchmark show that FLEET outperforms state‑of‑the‑art methods and remains robust across different observation frequencies.
The paper introduces QEF-GT-AdamW, a communication‑efficient and outage‑resilient algorithm for decentralized wireless federated learning. It combines gradient tracking, AdamW adaptive optimization, and dual‑stream biased quantization with error feedback to reduce communication payloads while mitigating non‑IID data effects. The method includes a local fallback strategy for unreliable broadcasts and provides convergence guarantees under compressed, unreliable wireless communication, demonstrating improved robustness and accuracy‑communication trade‑offs on heterogeneous MNIST and CIFAR‑10 datasets.
The paper introduces SPALT, a method that models spatio‑temporal locality for multi‑step forecasting of geo‑referenced time series. SPALT uses linear model trees to group series with similar trends, injecting spatial features locally, and employs a Reduced Error Pruning strategy that respects spatio‑temporal locality. Experiments on three real‑world renewable‑energy datasets show SPALT outperforms both tree‑based models and state‑of‑the‑art neural networks in forecasting energy production at multiple horizons.
MeMark introduces a watermarking scheme for Spiking Neural Networks that embeds a multi‑bit identifier directly into the membrane state of selected Leaky Integrate‑and‑Fire neurons, rather than in the output head. The watermark is recoverable by comparing neuron firing thresholds, eliminating the need for a learned decoder. Experiments on various SNN architectures—including a 215.4M‑parameter SpikeGPT checkpoint—show that all 20 independent 64‑bit keys reliably pass verification under a 51/64 rule, remain robust after fine‑tuning, pruning, quantization, and output‑head replacement, and are not recovered by random keys or adaptive attacks within the tested threat model.
The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.
Physics-informed neural networks (PINNs) often use over‑parameterized models, raising questions about necessary capacity and effective pruning. This study introduces physics‑informed spectrum‑aware pruning (PI‑SAP), which assigns saliency based on PDE residual sensitivity, and compares it to standard NTK‑SAP. Experiments on several nonlinear PDEs show PI‑SAP consistently preserves residual fidelity and remains competitive under aggressive sparsity, though no single criterion is optimal across all equations or sparsity levels.