arXiv Machine Learning
Aug 4
One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
arXiv:2608. 02091v1 Announce Type: new Abstract: A bfloat16 transformer can train normally for many steps and then collapse abruptly.
One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse
Related stories
arXiv Machine Learning
Sep 29
The Residual Stream's Effective Depth
The paper introduces effective depth (Deff), a scalar diagnostic that treats a transformer’s layer‑wise residual stream as a discrete‑time process and measures how representation similarity decays with layer distance. Across sixteen decoder‑only language models, Deff reveals that most models exhibit a lower similarity decay than the closed‑form reference, indicating correlated residual updates rather than unused depth. The study also shows that this effect is robust to various controls and persists early in training, suggesting Deff is a global accumulated‑state diagnostic rather than a capability score.
arXiv Computation and Language
Sep 28
Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness
arXiv:2609.30820v1 Announce Type: cross Abstract: Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes...
arXiv Machine Learning
Sep 2
Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance
The paper evaluates the effectiveness of tolerance‑based conformance tests for INT8 quantized GEMM kernels used in large language models. By injecting nine faults into a Qwen3‑1.7B reference pipeline, the authors show that most faults shift outputs by at most one bfloat16 spacing, rendering a tolerance of one spacing blind to these errors. They further demonstrate that requantizing weight scales to the nearest power of two aligns CUTLASS and Triton implementations bit‑for‑bit and produces identical token sequences, with only minor perplexity changes.
arXiv AI
Jul 3
Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map
arXiv:2607. 01854v1 Announce Type: cross Abstract: Can a platform tell, before deployment, whether an open-weight checkpoint has had its refusal mechanism stripped?
arXiv AI
Aug 26
Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
The study investigates why small language model agents tend to repeat a tool call that just failed. By recording the failed call and its error message in the transcript, the authors measure a negative corrective gain—agents are more likely to repeat the failed action, with a drop of about 1.03 nats per token. The problem is traced to the harness design rather than the model’s understanding of errors, and the authors show that replacing the verbatim call with a runtime-generated description of the failure can reduce this backfiring effect by 76%.
arXiv Computer Vision
Oct 2
Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null
arXiv:2610.00024v1 Announce Type: new Abstract: Across three vision-language model architectures (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3-8B), we report a universal negative finding for mid-layer inte...
Hugging Face Trending Papers
Sep 28
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
The paper investigates post‑training quantization of transformer attention blocks by optimizing a joint loss over the Q, K, V projections rather than individual weight matrices. Using this joint attention‑based objective (JAB), the authors achieve significant compression on Mistral‑7B, recovering 77‑90% of the performance gap at 3 bits, but the method fails when MLP layers are included. A role‑aware offset rule that ignores sensitivity estimates outperforms JAB on GPT‑2 and full Mistral‑7B, demonstrating that the matrix a weight belongs to is more critical than sensitivity metrics.
arXiv Machine Learning
Aug 4
Real-Time Detection and Repair of LLM Agent Failures
arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
arXiv AI
Aug 5
Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
arXiv:2608. 03222v1 Announce Type: cross Abstract: Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates.
arXiv Machine Learning
Sep 25
LastOPD: Taming Collapse in Latent On-Policy Distillation
The paper introduces LastOPD, a method that mitigates collapse in latent on‑policy distillation by applying latent supervision only to the last‑layer state during a brief cross‑fade into token‑level OPD. Experiments show that LastOPD improves MATH‑500 accuracy by 5.55 and 4.02 points over token‑only OPD when distilling Qwen3‑4B and Qwen3‑8B into Qwen3‑1.7B‑Base, and achieves comparable final scores in roughly half the training steps.
arXiv Machine Learning
Aug 5
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
arXiv:2608. 02689v1 Announce Type: cross Abstract: We convert 21 of 28 full-attention layers of Qwen3-0.