arXiv:2607.00171v2 Announce Type: replace
Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover on...
By Andrianos Michail, Stylianos Psychias, Michelle Wastl, Simon Clematide, Rico Sennrich, Juri Opitz
The paper proposes a multilingual adaptation of an English quality classifier to select high‑quality training data for large language model pretraining across more than 100 languages. The method trains a small multi‑layer perceptron on Transformer embeddings, using machine‑translated text scored by the English classifier as labels. Experiments on 1B, 3B, and 8B parameter models show that the adapted classifier preserves benchmark performance and regional knowledge, and it generalizes to unseen languages by learning the original scoring criteria.
By Vinko Sabol\v{c}ec, Bettina Messmer, Yassine Turki, Martin Jaggi
Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.
The study compares an English-only and a bilingual decoder-only model, each 310 M parameters, trained on eight diverse languages while controlling for English exposure, compute, and document overlap. After aligning on shared English vocabulary, the authors find that token embeddings appear similar, but the deeper hidden states used for prediction diverge across models. This hidden‑state mismatch grows through middle transformer layers and persists despite controls, indicating that contextual processing differs between the models.
"whyItMatters":"The findings show that embedding alignment can conceal significant internal representation differences, which is crucial for any downstream work that assumes aligned multilingual models are interchangeable."
By Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos
The paper challenges the assumption that improving cross‑lingual alignment automatically enhances cross‑lingual transfer. Using XLM‑R models aligned on token, sentence, and masked‑language‑modeling objectives across four language pairs, the authors evaluate zero‑shot transfer on part‑of‑speech tagging and sentence classification. They find that embedding‑based alignment metrics poorly predict downstream performance and that alignment and task gradients are often nearly orthogonal, especially when operating at different representational levels.
By Yana Veitsman, Yihong Liu, Hinrich Sch\"utze
The paper investigates cross‑lingual transfer for sequential sentence classification (SSC) in research papers, focusing on 13 non‑English languages. Experiments show that linguistic proximity does not reliably predict transfer success, whereas structural similarity in rhetorical organization—particularly label distribution similarity—correlates positively with performance. The authors introduce three generative‑model methods that exploit structural cues, achieving parity with strong encoder baselines on‑domain and outperforming them when transferring to unseen languages.
By Kazuhiro Yamauchi, Marie Katsurai