arXiv AI
Jul 28
Variational-Ising-Attention (VIA):TailoredAttentionMattersfor Science
arXiv:2607. 23634v1 Announce Type: cross Abstract: Attention enables context modeling via query-key scoring with softmax normalization.
Variational-Ising-Attention:Tailored Attention Matters for Science
Related stories
arXiv Machine Learning
Jun 26
Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention
arXiv:2603. 14717v2 Announce Type: replace Abstract: Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples.
arXiv Machine Learning
Oct 21
Interdomain Attention: Beyond Token-Level Key-Value Memory
The paper introduces Interdomain Attention (IA), a hybrid architecture that blends softmax attention with recurrent memory within a single layer, leveraging a kernel regression perspective to compress keys and values into a state‑space model (SSM) memory. IA reconstructs the attention matrix implicitly from memory states, preserving the computational efficiency of SSMs while achieving performance gains. Experiments on models ranging from 125 M to 1.3 B parameters show IA surpasses an S4D baseline at matched memory size, outperforms standard Transformers on perplexity and common‑sense reasoning, and matches or exceeds SSM‑based models on genomic downstream tasks.
arXiv Machine Learning
Sep 4
SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign
SimpleDesign is a single-stage, end-to-end model for joint protein sequence and structure design that eliminates the need for multi-stage training. It combines discrete cross-entropy for sequences with a regression objective for structures, using a Mixture-of-Transformer architecture to handle modality-specific processing while maintaining global self-attention. Trained on over 2 million sequence-structure pairs, SimpleDesign achieves strong performance on co-design and unconditional generation benchmarks.
arXiv Machine Learning
Sep 25
Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
The paper introduces Higher-Order Modular Attention (HOMA), a new attention mechanism that combines standard pairwise self‑attention with an explicit triadic attention pathway. HOMA uses overlapping blocks, local windows, and a low‑rank projection to make triadic interactions tractable. Experiments on controlled PARITY and MATCH3 tasks, as well as TAPE benchmarks, show that HOMA matches or outperforms matched pairwise and purely triadic baselines, especially when dependencies extend beyond triadic order, and it often converges faster and uses parameters more efficiently.
arXiv AI
Jun 8
Limitations of Normalization in Attention Mechanism
arXiv:2508. 17821v3 Announce Type: replace-cross Abstract: This paper investigates the limitations of the normalization in attention mechanisms.
arXiv AI
Jun 4
Retrieval and competition: how a protein foundation model starts a protein
arXiv:2605. 16331v2 Announce Type: replace-cross Abstract: Protein language models are increasingly used to guide experimental and clinical decisions, yet it is often unclear whether a confident prediction reflects recognition of biological evidence or retrieval of a statistical default.
arXiv AI
Jul 28
LC-SEPLM: long-range contact-supervised adaptation for sequence-only protein representation learning
arXiv:2607. 22777v1 Announce Type: cross Abstract: Protein language models learn transferable sequence representations.
arXiv AI
Jul 28
What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation
arXiv:2607. 22781v1 Announce Type: cross Abstract: High task performance does not show whether a model retains prediction-relevant structural information in its internal representation.
arXiv Machine Learning
Jul 15
PRIME: Protein Representation via Physics-Informed Multiscale Equivariant Hierarchies
arXiv:2605. 01625v3 Announce Type: replace Abstract: Proteins are inherently multiscale physical systems whose functional properties emerge from coordinated structural organization across multiple spatial resolutions, ranging from atomic interactions to global fold topology.
arXiv Machine Learning
Jun 17
Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.
arXiv Machine Learning
Aug 20
Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference
Monroe is a new molecular foundation model that improves upon existing models by pre‑training on over 81 million molecules from the PM6 quantum chemistry dataset, enhancing stereochemistry representation, and introducing novel training losses such as conformer denoising and embedding decorrelation. It also incorporates a prior‑data‑fitted model (TabPFN) for downstream in‑context prediction and demonstrates superior performance on Polaris benchmarks and activity cliff tests. Ablation studies show that the PFN‑based downstream approach can upgrade other models, producing state‑of‑the‑art variants MiniMol_PFN and CheMeleon_PFN.