← Back to all news
arXiv AI By Chaofei Hong, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang

AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation

Read the original on arXiv AI →

AC-JEPA is a plug‑in for supervised fine‑tuning that predicts masked intermediate states in both directions of a cross‑modal translation task, combining a joint‑embedding objective with causal token‑level supervision. It is designed to be robust to imperfect paired data by learning from continuous latent targets derived from pseudo‑labels, rather than hard categorical supervision. In optical chemical structure recognition, AC‑JEPA outperforms plain supervised fine‑tuning, achieving 76.07 % accuracy on clean pairs and 80.23 % when self‑training with pseudo‑labels, whereas SFT alone drops to 63.47 % under the same conditions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 16

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.

By Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif
llmsreinforcement-learningmultimodalbenchmarkssafety
More like this →
arXiv Computation and Language
Oct 12

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.

By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
llmsmultimodalbenchmarkssafety
More like this →
arXiv Machine Learning
Jun 10

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

arXiv:2606. 11190v1 Announce Type: new Abstract: Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality.

By Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets, Randall Balestriero
multimodalbenchmarkssafety
More like this →