arXiv AI

Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian

arXiv:2606. 21876v2 Announce Type: replace-cross Abstract: The Categorical Jacobian of Zhang et al.

arXiv AI

Aug 5

Cross-Layer Interaction under Weight-Space Ablation: A Closed-Form Attention Jacobian Bound and a Test on a Real Pretrained Model

arXiv:2608. 03629v1 Announce Type: new Abstract: A companion paper studies when activation patching and weight-space ablation agree, inside an idealized model where a conditional computation is carried additively through a residual stream.

By Abdallah Khemais

llms benchmarks

arXiv AI

Jun 2

DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models

arXiv:2606. 00091v1 Announce Type: cross Abstract: Joint Embedding Predictive Architectures (JEPAs) have reshaped self-supervised representation learning in vision.

By Sangdae Nam

llms rag diffusion fine-tuning benchmarks

arXiv Machine Learning

Jun 5

The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models

arXiv:2606. 05169v1 Announce Type: new Abstract: We give a stereological theory of LLM benchmark coverage.

By Jason Z Wang

llms benchmarks

arXiv Machine Learning

Jun 5

Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models

arXiv:2606. 05378v1 Announce Type: new Abstract: We test whether a single screen-and-ablate recipe -- identify attention-head circuits by task-pattern selectivity, then verify by causal ablation against a matched-random null -- produces consistent mechanistic claims across model families.

By Yongzhong Xu

llms benchmarks

arXiv AI

Aug 5

A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation

arXiv:2608. 03620v1 Announce Type: cross Abstract: Activation patching and weight-space ablation both claim a component is causally responsible for a behavior, yet they act on different objects: one forward pass versus the parameters behind every forward pass.

By Abdallah Khemais

llms benchmarks

arXiv AI

Jun 10

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

arXiv:2606. 10066v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years, yet reported accuracy assumes these examples were absent from pretraining.

By Bruce Changlong Xu, Lan Wu, Alexander Ryu

llms multimodal benchmarks

arXiv AI

Aug 5

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.

By Mohsen Arjmandi

llms rag benchmarks

arXiv Machine Learning

Jun 26

The Geometry of Updates: Fisher Alignment at Vocabulary Scale

arXiv:2606. 27242v1 Announce Type: new Abstract: Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets.

By John Sweeney

llms rag nlp safety

arXiv AI

Jul 17

Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect

arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?

By Ely Hahami, Ishaan Sinha, Lavik Jain

llms fine-tuning benchmarks safety

arXiv AI

Jul 24

Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.

By Sagar Dangal, Manoj Shakya

llms safety

arXiv AI

Jul 2

Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

arXiv:2607. 01002v1 Announce Type: cross Abstract: In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them.

By Aryo Pradipta Gema, Beatrice Alex, Pasquale Minervini

llms benchmarks

arXiv AI

1d ago

Exact Attention Sensitivity and the Geometry of Transformer Stability

arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.

By Seyed Morteza Emadi

llms

Related stories