arXiv:2607. 14661v1 Announce Type: new Abstract: Deploying large language models (LLMs) as personal assistants on mobile devices demands privacy, low latency, and offline availability, yet the computational cost of giant models clashes with strict edge-hardware budgets.
By Zhihan Jiang, Meng Li, Shenghao Liu, Keran Li, Ruiben Zhou, Xianjun Deng, Shuai Wang, Haipeng Dai
The paper introduces CERTA, a decision‑level statistical certificate that decides when a quantized clinical language model can replace its native BF16 reference and escalates uncertain decisions to the reference model. Across eleven instruction‑tuned models (7‑72 B parameters) and 2,500 clinical‑verification decisions from 2,484 de‑identified patients, 8‑bit quantization achieved 99.8‑100% coverage, while 4‑bit configurations cut deployment costs to roughly 26–28% of the BF16 baseline with peak memory below a 48 GB accelerator. On external named‑entity benchmarks, the chosen NF4 settings showed only a 0.045 mean absolute F1 change, indicating minimal performance loss.
whyItMatters":"CERTA enables risk‑controlled deployment of smaller, cheaper models for clinical decision support, potentially making advanced language models more accessible in resource‑constrained healthcare settings."
By Ali H. Lazem, William Teahan
ThinkBit is a two‑bit scalar quantization framework designed to preserve reasoning capabilities in language models. It assigns each layer a ternary, 2‑bit, or 3‑bit tier based on sensitivity, then uses a Gumbel‑softmax search to find optimal codewords block‑by‑block. After freezing codewords, it recalibrates shared scales via knowledge‑distillation and performs a brief quantization‑aware training, finally selecting the best checkpoint with paired reasoning tests.
By Rhea Sanjay Sukthanker, Pashmina Cameron
IronLLM-0.6B is a 654‑million‑parameter language model engineered for efficient on‑device inference, featuring a hybrid attention architecture, X‑MTP multi‑token prediction, and a lightweight verification head that yields a 1.48× decoding speedup. Trained on roughly 6.2 trillion tokens with a quality‑oriented pipeline and further refined via Multi‑Domain On‑Policy Distillation, the model adopts an Instruct‑Only design to meet low‑latency requirements. A lighter variant, IronLLM‑0.6B‑Light, replaces RMSNorm with Dynamic Tanh and streamlines costly components to enhance inference and quantization efficiency, offering a strong performance‑efficiency trade‑off for resource‑constrained deployment.
By Changdi Yang, Fengquan Jiao, Haochih Lin, Haoran Yang, Jing Xiao, Liangyu Huo, Suxin Lu, Tiance Chen, Wei Liu, Yinggan Xu, Yunxiang Lu, Zai Zheng, Zhirui Xie, Zhongyang Che, Ziyan Tang, Zuoxiang Zhao, Jian Yao
arXiv:2610.33839v1 Announce Type: new
Abstract: De-identifying clinical notes with a hosted language model discloses the information the step is meant to protect. We present EdgeDeID, a 0.6B-paramete...
By Sina Daneshgar, Raya Namazi, Amirali Khosravi, Pouria Rouzrokh, Bardia Khosravi
HyCo is a hybrid drafting method that assigns depth to a server model’s multi‑token prediction head for execution on a mobile NPU, while alternative branches are handled by a small language model in a single pass. By returning only the hidden state, input embedding, and key‑value entries, HyCo reduces server GPU time per token by 38.6% on Spec‑Bench and 29.2% on 32K‑token summarization, and achieves faster delivery and higher savings than an SLM‑only drafter. The approach also incorporates quantization, a restricted vocabulary, a 1K‑token window, and an ε‑threshold rule to relax exact sampling distribution preservation.
By Hanbin Bae, Heejin Choi, Joohyung Lee, Byoung Jin Choi, Sangjun Park, Hoon-Young Cho