ProtModernBERT is a modernized BERT-style protein language model that reduces training cost by incorporating architectural refinements, achieving performance comparable to larger models while cutting pre‑training FLOPs by 80‑90%. The authors benchmark ProtModernBERT against ProtBert, Amplify, and ESM2 on supervised structural tasks and a zero‑shot functional benchmark, showing it matches or closely tracks Amplify‑120M and ProtBert. Their cluster‑based validation and random‑init experiments reveal that modern architectural components provide stronger inductive biases than parameter count in low‑compute regimes.
Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especial...
ProtLingo is a protein language modeling framework that enhances a pretrained single‑sequence Transformer backbone with conditional local memory and sparse expert routing. It maps residue representations into discrete codes, composes local windows into latent N‑gram addresses, and retrieves reusable residual signals for recurring sequence contexts. The model also converts selected feed‑forward blocks into sparse Mixture‑of‑Experts layers, allowing residue‑dependent computation while activating only a subset of parameters, achieving competitive performance on protein fitness prediction, FLIP benchmarks, and supervised contact prediction with a 150M‑parameter backbone.
The study evaluates 4‑bit quantization and low‑rank adapter fine‑tuning (QLoRA) on several large protein language models, finding that many model‑task pairs retain over 90% of full fine‑tuning performance while achieving up to 90% GPU memory savings. QLoRA preserves early‑layer representations and induces task‑specific changes in later layers, closely resembling full fine‑tuning with smaller representational shifts. For generative models, 4‑bit quantization largely maintains structural and sequence‑level properties, though token‑level analysis reveals model‑dependent changes in autoregressive output distributions.
The paper introduces Prompteina, an all‑atom protein structure generation model that accepts natural‑language descriptions of desired structural and functional properties, similar to text‑to‑image systems. It is trained on a large paired dataset of 1.8 million protein structures and over 50 million captions, and fine‑tuned from the Proteina‑Complexa model to align residues with textual cues. Prompteina outperforms existing baselines on most structural metrics, improves binder design performance, and experimentally validates nine new binders for the GM2A target.
arXiv:2608. 12090v1 Announce Type: new Abstract: Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology.
arXiv:2603.29529v2 Announce Type: replace-cross Abstract: Using a statistical mechanics framework, we investigate the parameter space of transformer models trained on protein sequence data. We sample...
arXiv:2609.37675v1 Announce Type: new Abstract: Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and str...
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs).
arXiv:2610.19592v1 Announce Type: new Abstract: The ability to generate proteins with desired functions is a central goal of computational biology. Although natural language can express diverse desig...
The ability to generate proteins with desired functions is a central goal of computational biology. Although natural language can express diverse design objectives, existing protein generation methods...
arXiv:2505. 20346v3 Announce Type: replace-cross Abstract: Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering.
arXiv:2607. 22777v1 Announce Type: cross Abstract: Protein language models learn transferable sequence representations.