arXiv Machine Learning

Numerical Fragility in Transformers: A Layer-wise Theory for Risk Estimation and Selective Stabilization

arXiv:2510. 21770v2 Announce Type: replace Abstract: Low-precision execution can induce substantial forward discrepancies in Transformers even for fixed weights and input, yet these discrepancies are usually monitored only at the output and lack a layer-wise theoretical account.

arXiv AI

Jul 21

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.

By Irina Piontkovskaia, Sergey Nikolenko

llms fine-tuning

arXiv Machine Learning

Jul 21

An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers

arXiv:2607. 17696v1 Announce Type: cross Abstract: We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers and separate unconditional analytic results from conditional boundary-shape conclusions.

By Cheng Huan, Hongwei Yuan

llms safety

arXiv Machine Learning

Jul 21

Auditing Invisible Weight Updates with Reference Traces

arXiv:2607. 09800v2 Announce Type: replace Abstract: Master weights and stochastic rounding bypass invisible stored-weight updates but do not locate lost direct-storage proposals or parameters worth protecting.

By Zekai Shang

llms

arXiv Machine Learning

Aug 4

Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors

arXiv:2608. 00675v1 Announce Type: cross Abstract: Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.

By Alexander Scheinker

diffusion benchmarks

arXiv AI

Jul 15

Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter

arXiv:2607. 10203v2 Announce Type: replace-cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.

By Achyuthan Sivasankar

benchmarks

arXiv Machine Learning

Jun 19

Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

arXiv:2606. 19491v1 Announce Type: new Abstract: Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes.

By Tejas Pradeep Shirodkar, P. J. Narayanan

llms

arXiv Machine Learning

Jul 30

Dynamic Parameterization Is Not Dynamic Inference

arXiv:2607. 26192v1 Announce Type: new Abstract: Input-dependent controller coefficients are often treated as evidence of dynamic inference or computational savings.

By Zongfei Li, Yuan-yih Shang, Guozhong Luo

llms

arXiv Machine Learning

Jul 8

Heckman-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting

arXiv:2607. 05806v1 Announce Type: new Abstract: Training data for machine learning is routinely collected by a selection process the model never sees: loans are observed only when granted, outcomes only when a test was ordered.

By Gunner Levi Howe

safety

arXiv Machine Learning

Jul 1

Signed-Permutation Coordinate Transport for RMSNorm Transformers

arXiv:2606. 31963v1 Announce Type: new Abstract: Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, and merge alignments.

By John Sweeney

llms fine-tuning safety

arXiv Machine Learning

Aug 5

Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.

By Ravi Satya Durga Prasad Yenugula

llms

arXiv AI

Aug 5

A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation

arXiv:2608. 03620v1 Announce Type: cross Abstract: Activation patching and weight-space ablation both claim a component is causally responsible for a behavior, yet they act on different objects: one forward pass versus the parameters behind every forward pass.

By Abdallah Khemais

llms benchmarks

arXiv AI

Jul 14

When Does Depth Survive Composition? Compute--Quality Regimes in Latent World Models

arXiv:2607. 10203v1 Announce Type: cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.

By Achyuthan Sivasankar

benchmarks

Related stories