← Back to all news
arXiv Machine Learning By Jorge Castillo Sep\'ulveda, Marco Torres Y\'evenes, Juan Carlos Lanas

System-Prompt Conditioning and Hidden-State Geometry in Four Open-Weight Models: Corrections and What Survives

Read the original on arXiv Machine Learning →

The paper corrects earlier claims about how system prompts affect hidden‑state geometry in open‑weight language models. After auditing the code, the authors withdraw regime‑specific and direction‑to‑magnitude claims, but find that centroid distance, maximum mean discrepancy, and a linear probe still distinguish prompt conditions across all models. Curvature statistics only exceed noise in a minority of comparisons, and in Gemma‑4‑E4B‑it the identity prompt yields a lower‑norm state than a token‑matched generic prompt, aligning with observed pause behavior in outputs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 14

From Direction to Magnitude: How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States

arXiv:2607. 09842v1 Announce Type: new Abstract: We investigate whether identity-specifying system prompts produce statistically distinguishable geometric fingerprints in the hidden-state trajectories of four open-weight transformer language models spanning four post-training regimes: no training (Gemma-4-E4B base), multimodal RLHF (Gemma-4-E4B-it), RL distillation (DeepSeek-R1-Distill-Qwen-7B), and SFT (Qwen2.

By Jorge A. Castillo, Marco Torres Y\'evenes, Juan Carlos Lanas
llmsreinforcement-learningfine-tuningefficiencymultimodal
More like this →
arXiv Computation and Language
Aug 24

Prompt-Model Interaction Reaches the Fixed Points: A deterministic, task-free structural readout -- and the factorizations of it that failed

The paper demonstrates that a prompt’s influence is not inherent to the prompt itself but depends on the model, as prompts optimized for one model degrade on another and rankings shift under neutral reformatting. By examining a task‑free structural readout—specifically the fixed‑point behavior of a short‑window argmax map—the authors show that nine tokens of conditioning can move the fixed‑point fraction across most of its range, altering structural classes and model rankings, while instruction tuning has no effect. Attempts to explain this phenomenon through prefix length, content type, bidirectionality, or attention‑sink dominance all fail, indicating that the prompt‑model pair is the fundamental unit of explanation. whyItMatters":"The study reveals that prompt effectiveness is model‑specific and that simple structural readouts can capture this interaction, challenging assumptions about prompt generality and guiding future prompt‑engineering efforts."

By Nicol\'as Vera Z\'u\~niga
fine-tuning
More like this →
arXiv Machine Learning
Aug 27

Activation-Space Order-Swap Geometry: A Site-Asymmetry Audit

The paper introduces a site-asymmetry audit for activation-space interventions, decomposing order-dependent activation statistics into a canonical additive response from single interventions and an antisymmetrized second difference that removes first-order and self-curvature effects. Across six language-model families, the single-intervention baseline accounts for most of the bracket norm, while the corrected residual often clears a generic-interaction null. The method also transfers to non-language models, demonstrating its portability.

By Anqi Peter Li
More like this →