← Back to all news

Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

5,914 stories · RSS feed

arXiv AI
Sep 16

ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training

ReDraft is a reference‑driven revision method for continual post‑training of large multimodal language models. It uses the model’s own incorrect outputs as references, revises them, verifies the revisions, and fine‑tunes on the accepted ones, thereby combining explicit supervision with policy proximity. On tasks such as Counting, Clock Reading, and Jigsaw, ReDraft outperforms standard supervised fine‑tuning and on‑policy methods, achieving higher target‑task gains while dramatically reducing forgetting.

By Zhihao Zhang, Mingqi Wu, Qiaole Dong, Enyu Zhou, Shuo Li, Boyang Liu, Jiazheng Zhang, Honglin Guo, Xin Guo, Shaofan Liu, Junzhe Wang, Dingwei Zhu, Zhiheng Xi, Minlong Peng, Yuan Hua, Qi Zhang, Tao Gui, Xuanjing Huang
fine-tuningefficiencymultimodal
More like this →
arXiv Machine Learning
Sep 16

EBL: Efficient Broad Learning for Distributed Adaptive Harmonic Analysis

The paper introduces EBL, an Efficient Broad Learning framework designed for distributed adaptive harmonic estimation in power grids affected by electric vehicle charging. It leverages a quantised FPGA implementation to provide high‑accuracy, half‑cycle input harmonic predictions with ultra‑low latency, outperforming existing FPGA methods by 17.4×. The online transfer learning component enables rapid adaptation across multiple charging scenarios, while bespoke quantisation and sparsity reduce resource usage to just 5.9% of the LUTs on a Zynq Ultrascale+ FPGA, compared to 82% of the state‑of‑the‑art accelerator.

By Changhong Li, Georgios Floros, Biswajit Basu, Shreejith Shanker
fine-tuningefficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 16

Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching

The paper introduces Window-Diffusion, a method that accelerates diffusion language model inference by pruning and caching tokens within a sliding window. It categorizes undecoded tokens into active, buffer, and far-field groups, computing only the first two while discarding the rest. Experiments on LLaDA and Dream demonstrate up to 99× speedup with minimal loss in generation quality.

By Fengrui Zuo, Zhiwei Ke, Yiming Liu, Wenqi Lou, Chao Wang, Xuehai Zhou
llmsdiffusionefficiency
More like this →
arXiv AI
Sep 16

Large Language Models in the Loop: A Stability- and Network-Aware Survey in Networked Control, Cyber-Physical, and Multi-Agent Systems

The article surveys how large language models (LLMs) can be incorporated into networked control systems, cyber‑physical systems, and multi‑agent networks without violating stability and safety guarantees. It proposes treating the LLM as a slow supervisor that sets high‑level goals, while a fast, certified inner loop preserves physical stability. The survey maps LLM characteristics—such as inference latency, API failures, tokenization, and hallucinations—to classical control challenges and highlights the growing gap between model capability and formal safety assurances, calling for future research on stability proofs.

By Haiping Du, Linping Chan
llmsagentsnlpefficiencysafety
More like this →
arXiv AI
Sep 16

Verbalizing Subliminal Learning Effects Using Text Optimization

The paper introduces SALVE, a method that uses text optimization to uncover and verbalize subliminal learning effects in language models. By optimizing a soft prompt and converting it into a legible text prompt, SALVE can reliably recover the teacher model’s hidden traits that are transmitted through a distillation dataset. The authors demonstrate SALVE’s effectiveness across various scenarios, including mixed datasets, biased teacher activation, and preference‑selected data, thereby providing a tool for detecting hidden influences in model training.

By Nathan Hu, Sanmi Koyejo, Christopher Potts
efficiency
More like this →
arXiv AI
Sep 16

OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation

OPD‑Aha is a privileged on‑policy distillation method that improves multimodal reasoning by reconstructing the distillation target from the teacher’s isolated visual preference instead of relying on fragile teacher‑student discrepancies. It suppresses continuations that contradict the image, encouraging students to interrupt flawed reasoning with reflection tokens such as "wait" and "actually." This approach leads to consistent improvements across fine‑grained perception and complex multimodal reasoning benchmarks.

By Chenhao Qiu, Dawei Li, Yechao Zhang, Lei Gong, Zhen Tan
efficiencymultimodalbenchmarkssafety
More like this →
arXiv AI
Sep 16

QueryFormer: Winning Solution for KDD Cup 2026 Tencent UniRec Challenge

QueryFormer is a unified architecture designed for post‑click conversion rate prediction, addressing both feature interactions and sequential user behaviors. It introduces a stackable field–sequence block that generates query tokens via cross‑attention and packs sequence queries into shared‑parameter attention, improving efficiency and accuracy. The model won first place in the KDD Cup 2026 Tencent UniRec Challenge Industrial Track with an AUC of 0.83254, and scaling studies show that increasing view width slightly boosts validation AUC while maintaining low latency.

By Yuanzhe Zhou, Zhaoyang Zeng
efficiencybenchmarks
More like this →
arXiv AI
Sep 16

SafeFlow: Real-Time Text-Driven Humanoid Whole-Body Control via Physics-Guided Rectified Flow and Selective Safety Gating

SafeFlow is a real‑time, text‑driven humanoid control framework that blends physics‑guided motion generation with a three‑stage safety gate. It uses Physics‑Guided Rectified Flow Matching in a VAE latent space to produce physically executable trajectories, accelerates sampling with Reflow, and filters unsafe outputs via semantic OOD detection, directional sensitivity checks, and hard kinematic constraints before handing them to a motion‑tracking controller. Experiments on the Unitree G1 show that SafeFlow achieves higher success rates, better physical compliance, and faster inference than diffusion‑ and retargeting‑based baselines while maintaining motion diversity.

By Hanbyel Cho, Sang-Hun Kim, Jeonguk Kang, Donghan Koo
ragdiffusionroboticsefficiencysafety
More like this →
arXiv Machine Learning
Sep 16

LCAP: Population-Informed Latent Chip Adaptation from Few Output Probes for Photonic Neural Networks

The paper introduces LCAP, a method for adapting photonic neural networks to real hardware by learning a shared correction from a population of chips and then personalizing each chip using only 32 fixed output probes. LCAP decomposes adaptation into a transferable population correction and a probe‑inferred latent personalization, allowing feed‑forward calibration without device‑specific optimization. Experiments on a simulated three‑layer 64‑mode MZI network show accuracy improvements from 80.4% to 93.4% and significant gains on unseen chips.

By Tianyu Gao, Guantian Zheng
roboticsefficiency
More like this →
arXiv Machine Learning
Sep 16

Adaptive Bayesian Partner Selection for Federated Clinical Centers

Adaptive Bayesian Partner Selection (ABPS) is a peer‑to‑peer federated learning framework designed for heterogeneous clinical centers, where each center maintains a Beta‑Bernoulli posterior over prospective peers’ Shapley marginal utility and selects partners using an Upper Confidence Bound criterion. The lightweight propose‑reject mechanism allows centers to collaborate only when mutually beneficial, with the option to abstain from communication entirely. In experiments on 230 non‑IID ICU centers predicting in‑hospital mortality, the ABPS‑X variant achieves comparable accuracy to the strongest baseline (FedDyn) while reducing communication cost by 90% and enabling intentional isolation for many centers.

By Navid Seidi, Satyaki Roy, Sajal K. Das
efficiency
More like this →