← Back to all news
arXiv Machine Learning By Alberto Mart\'inez, Natalia Flechas Manrique, Lloren\c{c} Espinosa-Portal\'es, Luc Andrea

ProtModernBERT: Compute-Efficient Protein Language Modeling

Read the original on arXiv Machine Learning →

ProtModernBERT is a modernized BERT-style protein language model that reduces training cost by incorporating architectural refinements, achieving performance comparable to larger models while cutting pre‑training FLOPs by 80‑90%. The authors benchmark ProtModernBERT against ProtBert, Amplify, and ESM2 on supervised structural tasks and a zero‑shot functional benchmark, showing it matches or closely tracks Amplify‑120M and ProtBert. Their cluster‑based validation and random‑init experiments reveal that modern architectural components provide stronger inductive biases than parameter count in low‑compute regimes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 7

ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing

ProtLingo is a protein language modeling framework that enhances a pretrained single‑sequence Transformer backbone with conditional local memory and sparse expert routing. It maps residue representations into discrete codes, composes local windows into latent N‑gram addresses, and retrieves reusable residual signals for recurring sequence contexts. The model also converts selected feed‑forward blocks into sparse Mixture‑of‑Experts layers, allowing residue‑dependent computation while activating only a subset of parameters, achieving competitive performance on protein fitness prediction, FLIP benchmarks, and supervised contact prediction with a 150M‑parameter backbone.

By Mingrui Li, Sixian Shen, Minzhang Li, Ruiyi Zhang, Kexin Zhang, Jiakai Zhang, Jingyi Yu
llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 3

Analysis of Quantized and Efficiently Adapted Protein Language Models

The study evaluates 4‑bit quantization and low‑rank adapter fine‑tuning (QLoRA) on several large protein language models, finding that many model‑task pairs retain over 90% of full fine‑tuning performance while achieving up to 90% GPU memory savings. QLoRA preserves early‑layer representations and induces task‑specific changes in later layers, closely resembling full fine‑tuning with smaller representational shifts. For generative models, 4‑bit quantization largely maintains structural and sequence‑level properties, though token‑level analysis reveals model‑dependent changes in autoregressive output distributions.

By Ilan Yaniv Zeisler, Sebastian Clancy, Pouriya Bayat, Saaim Raad, Ivan Kraskov, Matthew Xie, Vivian White, Spencer Perkins, Serena Singh, Sepehr Bayat, Keith Pardee
llmsfine-tuningefficiency
More like this →
arXiv Machine Learning
Oct 12

Designing Protein Structures from Natural Language

The paper introduces Prompteina, an all‑atom protein structure generation model that accepts natural‑language descriptions of desired structural and functional properties, similar to text‑to‑image systems. It is trained on a large paired dataset of 1.8 million protein structures and over 50 million captions, and fine‑tuned from the Proteina‑Complexa model to align residues with textual cues. Prompteina outperforms existing baselines on most structural metrics, improves binder design performance, and experimentally validates nine new binders for the GM2A target.

By Joseph Lee, Yanjing Li, Bo Qiang, Gregory Lau, Allyson Ettinger, Noah A. Smith, Rory Donovan-Maiye, David Baker, Hanna Hajishirzi, Pang Wei Koh
diffusionfine-tuning
More like this →