The paper introduces IDiom, an autoregressive protein language model trained on a large dataset of intrinsically disordered protein regions (IDRs) from AlphaFold, and demonstrates that it can generate sequences matching natural IDR composition, motifs, and disorder. It further presents RL‑SAE, a reinforcement learning approach that uses sparse autoencoder features to steer generation toward specific functional patterns, achieving high activation of targeted features and improved predicted subcellular localization and transcriptional activity. The combination of IDiom and RL‑SAE allows interpretable, composable IDR design by explicitly controlling function‑associated sequence features.
By Jason X. Liu, Sebastian Ibarraran, Frank Hu, Soojung Yang, Xinyu A. Feng, Abigail Park, Anagha Aneesh, Lacramioara Bintu, Alexander R. Dunn, Grant M. Rotskoff
arXiv:2606. 27440v1 Announce Type: new Abstract: Foundation models for structural biology have achieved remarkable performance in predicting biomolecular structure and show promise for the design of proteins and small molecules.
By Giosue Migliorini, Aristofanis Rontogiannis, Grigori Guitchounts, Nicholas Franklin, Axel Elaldi, Olivia Viessmann
The paper introduces a method that uses orthogonal projection to remove the influence of known biochemical features from protein language model (PLM) embeddings, allowing the authors to assess how much these features contribute to protein fitness predictions. By applying this technique to high‑order and interaction effects, they demonstrate that eliminating these interpretable features reduces downstream classifier performance, indicating that PLM embeddings encode patterns correlated with biochemical properties. The authors also show that these biochemical features explain a substantial portion of the variance in the classifier’s predictions, suggesting that PLM embeddings capture biologically relevant information.
By Paulo Yanez Sarmiento, Pia Francesca Rissom, Manuel Pfeuffer, Marco Simnacher, Jordan F. Safer, Sumaiya Iqbal, Henrike O. Heyne, Nadja Klein, Bernhard Y. Renard
arXiv:2605. 01625v3 Announce Type: replace Abstract: Proteins are inherently multiscale physical systems whose functional properties emerge from coordinated structural organization across multiple spatial resolutions, ranging from atomic interactions to global fold topology.
By Viet Thanh Duy Nguyen, John K. Johnstone, Truong-Son Hy
arXiv:2607. 22777v1 Announce Type: cross Abstract: Protein language models learn transferable sequence representations.
By Chen Wang, Boming Kang, Qinghua Cui
arXiv:2608. 12090v1 Announce Type: new Abstract: Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology.
By Roman Joeres, Ilya Senatorov, Olga V. Kalinina
SoftGene is a new framework that enhances gene set annotation by combining protein language model representations with a hybrid prompting scheme. It uses a hierarchical attention-based encoder built on ESM to encode gene sets from protein amino acid sequences, then merges soft prompts derived from these embeddings with hard prompts generated by a large language model. The approach is evaluated on Gene Ontology and MSigDB datasets, showing that integrating protein-sequence information with textual context improves overall annotation performance, though the benefit varies across biological domains.
By Drew Ross, Arya Hadizadeh Moghaddam, Dongjie Wang, Xiaoyu Zhang, Zijun Yao
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs).
arXiv:2605.16581v2 Announce Type: replace
Abstract: Masked language modeling (MLM) is the standard objective for training protein language models, typically implemented by randomly masking individual...
By Thomas Walton, Ayan Goel, Amirali Aghazadeh
ProtModernBERT is a modernized BERT-style protein language model that reduces training cost by incorporating architectural refinements, achieving performance comparable to larger models while cutting pre‑training FLOPs by 80‑90%. The authors benchmark ProtModernBERT against ProtBert, Amplify, and ESM2 on supervised structural tasks and a zero‑shot functional benchmark, showing it matches or closely tracks Amplify‑120M and ProtBert. Their cluster‑based validation and random‑init experiments reveal that modern architectural components provide stronger inductive biases than parameter count in low‑compute regimes.
By Alberto Mart\'inez, Natalia Flechas Manrique, Lloren\c{c} Espinosa-Portal\'es, Luc Andrea
MT-ProtBERT is a multi‑task extension of ProtBERT designed for classifying intrinsically disordered proteins (IDPs) in low‑data settings. It combines Dynamic Window Masking, a Multi‑Scale 1D Convolutional classifier, and auxiliary biochemistry‑informed objectives to jointly optimize masked language modeling and domain‑specific tasks. In experiments on phosphorylation site prediction and protein compaction prediction, MT‑ProtBERT outperforms the RNN‑based IDP model PARROT across all limited‑data tasks.
By Jian Sun, Kingshuk Ghosh, Lilianna Houston, Mohammad H. Mahoor
Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especial...