arXiv Machine Learning October 3, 2026 By Naveen Mysore

Attention Manifolds: Steering or Blocking Language Models by Editing Learned B-Spline Surfaces

Read the original on arXiv Machine Learning →

Attention Manifolds introduce learned 2D B-spline surfaces that modulate each value dimension in transformer attention based on query-key interactions. Applied to LLaMA models, they reduce WikiText-2 perplexity by 2–2.5 points with minimal parameter overhead and alter greedy-decoded outputs in 69–83% of diverse prompts, especially on ambiguous inputs. The surfaces can be edited mechanically to steer or block value flow, offering a geometric tool for improving output quality and potentially enhancing safety.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Oct 2

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning

The paper investigates which linguistic operations can be replaced by latent task representations instead of in‑context prompts. A 2.6 M‑parameter network learns to encode the geometry of a few‑shot support set and applies an additive update to a frozen GPT‑2 model, achieving zero‑shot performance comparable to or better than 10‑shot in‑context learning for forward inflection and lemmatization. However, for arbitrary pairings such as antonymy, the latent approach only reaches about half the performance of in‑context learning, indicating that productive rules amortize well while memorized pairings do not.

By Gunmay Jhingran

llms

arXiv Machine Learning

Oct 5

Hesitation Has a Geometry: Entropy-Trained Hyperbolic Probes for Sparse Activation Steering

arXiv:2610.02391v1 Announce Type: new Abstract: When a large language model solves a mathematical problem, its reasoning is largely hierarchical, and the solution often branches at a few tokens where...

By Zeyong Zhang, Tung Sum Thomas Kwok, Tengfei Ma, Mengjia Xu

llms

arXiv Computation and Language

Oct 1

Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models

arXiv:2410.02343v2 Announce Type: replace Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...

By Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov

llms rag nlp benchmarks safety

arXiv Computation and Language

Sep 15

Harmfulness Propagation Dynamics: Layer-wise Trajectories of Adversarial Intent in Large Language Models

arXiv:2609.13534v1 Announce Type: new Abstract: We identify \textbf{Harmfulness Propagation Dynamics (HPD)}: for harmful prompts, the projection of the last-token hidden state onto a learned harm dir...

By Noor Islam S. Mohammad, Ulu\u{g} Bayaz{\i}t

llms benchmarks safety

arXiv Machine Learning

Sep 15

Exact Finite Attention Responses From RoPE Derivatives

arXiv:2609.14127v1 Announce Type: cross Abstract: We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The...

By Julie Huang, Maggie Chlon, Gregory Gutin, Leon Chlon

arXiv AI

Jul 7

Unsupervised Features Mining via Activation Geometry

arXiv:2607. 04222v1 Announce Type: new Abstract: Interpretability methods aim to reveal the features represented inside large language models (LLMs).

By Amit LeVi, Elad David, Max Fomin

llms safety