← Back to all news
Simon Willison

Qwen 3.8 27B on my laptop can interpret model weights, too

Read the original on Simon Willison →

The article discusses a recent paper titled "Sparks of Weight‑Space Programming," which shows that advanced language models such as GPT‑6 Astra can read and modify the raw weights of neural networks. The authors demonstrate that these models can interpret what a network was trained to do, write new small game‑playing networks from scratch, and even improve the math performance of Qwen‑3.5‑0.8B by editing a tiny fraction of its weights. The author notes that Qwen 3.8 27B running on a laptop can perform similar weight‑space programming tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Simon Willison.

Related stories

Hugging Face Trending Papers
Sep 28

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers

The paper investigates post‑training quantization of transformer attention blocks by optimizing a joint loss over the Q, K, V projections rather than individual weight matrices. Using this joint attention‑based objective (JAB), the authors achieve significant compression on Mistral‑7B, recovering 77‑90% of the performance gap at 3 bits, but the method fails when MLP layers are included. A role‑aware offset rule that ignores sensitivity estimates outperforms JAB on GPT‑2 and full Mistral‑7B, demonstrating that the matrix a weight belongs to is more critical than sensitivity metrics.

llmsfine-tuningefficiency
More like this →
arXiv Machine Learning
Oct 28

Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State

The paper introduces an error‑diffusion (ED) optimizer that replaces the full‑precision latent copy of binary‑weight language models with a small 2‑8‑bit residual buffer, reducing training state per weight from 97 bits to 3‑9 bits—up to 19× less. ED accumulates unrealized weight updates in the buffer and flips the binary weight when the accumulated value crosses a threshold aligned with the quantization grid, enabling efficient training on large transformers with minimal loss compared to latent‑weight baselines. The method scales to models with up to 19 M parameters on enwik8 and 70 M on wikitext‑103, maintaining performance within a few percent of reference while allowing on‑device adaptation within a 275 MB memory budget.

By Pierre-Olivier Bonin
llmsragdiffusionefficiencybenchmarkssafety
More like this →