arXiv:2609.18078v4 Announce Type: replace
Abstract: Grokking, or delayed generalization after memorization, incurs substantial training steps across algebraic tasks. Cross-operator transfer from a gr...
By Zeyu Jia
arXiv:2606. 08635v1 Announce Type: new Abstract: Prefill-decode (PD) disaggregation decouples prompt processing from token generation, but it also turns the key-value (KV) cache into a network payload.
By Yang Pengju
The paper investigates converting a large pretrained transformer (1.4 B parameters) into a smaller sibling (410 M) by studying representation alignment and parameter projection. It finds that dense weight projection destroys structure, and that a low‑budget, structure‑aware compensation—separating least‑squares function alignment from variance‑preserving rescaling—yields significant gains on token‑efficient training, outperforming subcloning and standard distillation pipelines at matched budgets.
By Ravi Satya Durga Prasad Yenugula
arXiv:2608.30427v1 Announce Type: cross
Abstract: Speculative decoding speeds up generation with an efficient draft model (drafter) that proposes tokens for a target model to verify in one pass, pres...
By Ephrem Wu
arXiv:2607. 18553v1 Announce Type: cross Abstract: Can a language model read the quality of ongoing computation, and can an external intervention turn that readout into better outcomes?
By Jan Kirin
arXiv:2603. 28921v3 Announce Type: replace-cross Abstract: The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)).
By Ivan Pasichnyk
arXiv:2607. 09156v1 Announce Type: new Abstract: Additive activation steering (injecting a scaled residual-stream direction during generation) is calibrated almost entirely in single-turn chat, yet the models it targets are increasingly deployed as tool-using ReAct agents.
By Lucas Pinto
arXiv:2610.15509v1 Announce Type: cross
Abstract: KV caches make long-context serving efficient by reusing past computation, but model switching breaks this reuse and forces the receiver to prefill t...
By Haomiao Qiu, Yining Tang, Zijia Song, Ruizi Han, Ziyue Qiao, Miao Zhang
arXiv:2605. 07284v2 Announce Type: replace Abstract: A late-layer change learned during post-training may work on the base model's earlier state, or it may depend on earlier computation learned with it.
By Yifan Zhou
arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.
By Ravi Satya Durga Prasad Yenugula
The paper investigates whether sink-aware attention head selection remains valid after 4‑bit NF4 weight‑only post‑training quantization. Using Sink Topology Consistency metrics, it finds that global rank preservation stays high across Qwen2.5 and Llama‑3.2 models, yet top‑k head overlap drops to 61–79% and layer‑specific sink‑mass shifts can be substantial. The study also shows that cross‑domain calibration degrades more than within‑domain precision and that recalibration with a small number of samples can recover most of the stability, though full‑map stability may require updating more layers.
By Kuanlin Chen, Chen-Wei Kuo, Cheng-En Ou
arXiv:2606. 00382v1 Announce Type: new Abstract: Sequential fine-tuning of large language models forces a choice: let the shared substrate keep learning and accept catastrophic forgetting, or freeze it after task one and foreclose cross-task refinement.
By Kiran Nayudu, Aswini Nutakki, Sai Vinay Naidu, Ashwin Shanmugasundaram