arXiv Machine Learning

Entropy, Disagreement, and the Limits of Foundation Models in Genomics

arXiv:2604. 04287v2 Announce Type: replace Abstract: Foundation models in genomics have shown mixed success compared to their counterparts in natural language processing.

arXiv AI

Aug 12

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

arXiv:2602. 17162v3 Announce Type: replace Abstract: Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature".

By Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi, Nicole Bussola, Simon Lee, Shane O'Connell, Dung Hoang, Marissa Wirth, Alexander W. Charney, Nati Daniel, Yoli Shavit

rag diffusion benchmarks safety

arXiv Machine Learning

6d ago

Task- and dataset-specific information in protein language models

arXiv:2608. 12090v1 Announce Type: new Abstract: Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology.

By Roman Joeres, Ilya Senatorov, Olga V. Kalinina

llms rag nlp

Hugging Face Trending Papers

Aug 12

Task- and dataset-specific information in protein language models

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs).

llms rag nlp

arXiv Machine Learning

Jul 7

LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure

arXiv:2607. 04733v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities.

By Yueyang Wang, Baolong Bi, Shuo Lu, Jingyuan Zhang

llms fine-tuning multimodal

arXiv AI

Jul 13

TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology

arXiv:2607. 08803v1 Announce Type: cross Abstract: The push toward large language models for biology (BioLM) has created a need for training corpora that can endow models with a genuine understanding of biology.

By Hyunjin Seo, Hyeon Hwang, Gyubok Lee, Jay Shin, Jimin Park, Taesoo Kim, Sanghoon Lee, Hongjoon Ahn, Sungjun Han, Sangwon Jung

llms

arXiv AI

Jul 1

Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

arXiv:2410. 12341v4 Announce Type: replace-cross Abstract: As AI-generated content increasingly populates the web, generative AI models are at growing risk of being trained on their own outputs, a process known as AI autophagy.

By Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, Luca Pappalardo

llms fine-tuning

arXiv AI

Jul 23

Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models

arXiv:2607. 19618v1 Announce Type: cross Abstract: Genomic language models achieve strong performance across regulatory-genomics tasks, yet what these models internally represent remains opaque, and the field lacks a principled procedure for verifying that an apparent ``concept'' inside a model is real rather than an artifact of sequence composition.

By Sarwan Ali

llms nlp safety

arXiv AI

Aug 10

bioMoR: Biology-Guided Mixture-of-Recursions for Effective Genomic Learning

arXiv:2608. 06727v1 Announce Type: new Abstract: Transformer models for high-dimensional omics analysis process thousands of genes or pathways, although only a subset requires deep computation.

By Koushik Howlader, Tirtho Roy, Md Tauhidul Islam, Wei Le

llms rag benchmarks safety

arXiv AI

Jun 9

Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

arXiv:2606. 08417v1 Announce Type: cross Abstract: Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.

By Antonio Franca, Alexander Tong

llms diffusion benchmarks safety

arXiv Machine Learning

1d ago

Language models suffer from a curse of ambiguity

arXiv:2608. 15448v1 Announce Type: cross Abstract: Large language models increasingly rely on sampling as a driver of their own improvement, making the fidelity of their learned distributions more critical than ever.

By Nicolas Zucchet, Hyun Dong Lee, Scott Linderman

llms rag

arXiv AI

Jul 17

Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions

arXiv:2607. 14306v1 Announce Type: new Abstract: In this paper, we study the connection between an LLM's output distribution and the data used to train it.

By Zachary Izzo

llms safety

Hugging Face Trending Papers

Jul 6

LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure

Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard cross-entropy fine-tuning promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, potentially distorting the rich local preference structure learned during pretraining.

llms fine-tuning multimodal

Related stories