← Back to all news
Simon Willison

Qwen 3.8 27B on my laptop can interpret model weights, too

The article discusses a recent paper titled "Sparks of Weight‑Space Programming," which shows that advanced language models such as GPT‑6 Astra can read and modify the raw weights of neural networks. The authors demonstrate that these models can interpret what a network was trained to do, write new small game‑playing networks from scratch, and even improve the math performance of Qwen‑3.5‑0.8B by editing a tiny fraction of its weights. The author notes that Qwen 3.8 27B running on a laptop can perform similar weight‑space programming tasks.

Related stories

Hugging Face Trending Papers
Sep 28

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers

The paper investigates post‑training quantization of transformer attention blocks by optimizing a joint loss over the Q, K, V projections rather than individual weight matrices. Using this joint attention‑based objective (JAB), the authors achieve significant compression on Mistral‑7B, recovering 77‑90% of the performance gap at 3 bits, but the method fails when MLP layers are included. A role‑aware offset rule that ignores sensitivity estimates outperforms JAB on GPT‑2 and full Mistral‑7B, demonstrating that the matrix a weight belongs to is more critical than sensitivity metrics.

llmsfine-tuningefficiency
More like this →
arXiv Machine Learning
Oct 28

Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State

The paper introduces an error‑diffusion (ED) optimizer that replaces the full‑precision latent copy of binary‑weight language models with a small 2‑8‑bit residual buffer, reducing training state per weight from 97 bits to 3‑9 bits—up to 19× less. ED accumulates unrealized weight updates in the buffer and flips the binary weight when the accumulated value crosses a threshold aligned with the quantization grid, enabling efficient training on large transformers with minimal loss compared to latent‑weight baselines. The method scales to models with up to 19 M parameters on enwik8 and 70 M on wikitext‑103, maintaining performance within a few percent of reference while allowing on‑device adaptation within a 275 MB memory budget.

By Pierre-Olivier Bonin
llmsragdiffusionefficiencybenchmarkssafety
More like this →
arXiv Machine Learning
4d ago

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

The paper investigates the training‑inference mismatch that arises when large language models are trained with on‑policy reinforcement learning but inference is performed by a separate engine. It identifies two sources of mismatch—numerical discrepancies in the engine and policy lag due to stale weights—and shows that correcting one source can worsen the other. The authors propose a discrepancy‑lag correction (DLC) that centers numerical errors, clips lag ratios, and resynchronizes the sampler, achieving stable training on Qwen models.

By Yu Li, Yufa Zhou, Tian Lan
llmsreinforcement-learningsafety
More like this →
arXiv AI
Sep 11

Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution

The paper reports a large‑scale post‑training ternarisation of the Qwen3 language model, extending a conversion pipeline from the 4B to the 8B variant. Using KOTMS rotation, E2M‑ATQ adaptive ternarisation, and GPTQ‑style error compensation, the authors achieve a 1.361× perplexity ratio across three corpora and retain 78.5% of the FP16 accuracy on zero‑shot tasks, with the 8B model outperforming the 4B by 8.9 percentage points. The study also demonstrates lossless lattice‑aware packing, producing an 8.24 GiB checkpoint that preserves perplexity, and shows that direct packed execution can reach 15.52 tokens/s in 7.35 GiB, though packed GEMV remains slower than FP16 cuBLAS.

By Anirudh Malik, M Sparsh Mehra, Poojith Devan
llms
More like this →