arXiv AI
ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 12
JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures
arXiv:2602. 17162v3 Announce Type: replace Abstract: Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature".
arXiv Machine Learning
Sep 1
Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees
The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.
arXiv AI
Jun 6
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.
arXiv AI
Aug 21
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models
arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.
arXiv AI
2d ago
Cross-Lingual Alignment for Decoder-Only Models using MoE Routers
arXiv:2610.01921v1 Announce Type: cross Abstract: Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is...
arXiv AI
Jul 29
REPREC: Representation Driven Parameter-Efficient Recommendation System
arXiv:2607. 24845v1 Announce Type: cross Abstract: Large language models (LLMs) have been applied to sequential recommendation by formulating it as a natural language task.