arXiv Machine Learning
LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 12
JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures
arXiv:2602. 17162v3 Announce Type: replace Abstract: Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature".
arXiv AI
Jun 12
HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokens
arXiv:2512. 15133v3 Announce Type: replace-cross Abstract: Proteins inherently possess a consistent sequence-structure duality.
arXiv AI
Sep 7
ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing
ProtLingo is a protein language modeling framework that enhances a pretrained single‑sequence Transformer backbone with conditional local memory and sparse expert routing. It maps residue representations into discrete codes, composes local windows into latent N‑gram addresses, and retrieves reusable residual signals for recurring sequence contexts. The model also converts selected feed‑forward blocks into sparse Mixture‑of‑Experts layers, allowing residue‑dependent computation while activating only a subset of parameters, achieving competitive performance on protein fitness prediction, FLIP benchmarks, and supervised contact prediction with a 150M‑parameter backbone.
arXiv Machine Learning
Sep 1
EvoLen: Evolution-Guided Tokenization for DNA Language Model
EvoLen is a tokenization method for DNA language models that incorporates evolutionary information to prioritize functional sequence patterns such as regulatory motifs. It groups DNA sequences by cross-species evolutionary signals, trains separate BPE tokenizers for each group, merges vocabularies with a rule that favors preserved patterns, and uses length-aware decoding with dynamic programming. Experiments show EvoLen better preserves functional motifs, differentiates genomic contexts, and aligns with evolutionary constraints while matching or surpassing standard BPE on various DNALM benchmarks.
arXiv Machine Learning
Jun 26
Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attention
arXiv:2603. 14717v2 Announce Type: replace Abstract: Generating novel protein sequences that respect a family's statistical constraints typically requires training deep generative models on thousands to millions of examples.
arXiv Machine Learning
Jun 4
Towards A Generative Protein Evolution Machine with DPLM-Evo
arXiv:2605. 00182v3 Announce Type: replace Abstract: Proteins are shaped by gradual evolution under biophysical and functional constraints.