arXiv Computation and Language

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

arXiv Computation and Language

Sep 2

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

arXiv:2607.00171v2 Announce Type: replace Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover on...

By Andrianos Michail, Stylianos Psychias, Michelle Wastl, Simon Clematide, Rico Sennrich, Juri Opitz

rag nlp benchmarks

arXiv AI

Jul 7

Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

arXiv:2607. 04071v1 Announce Type: cross Abstract: Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world.

By Lucas Hideki Takeuchi Okamura, Alexandre Alcoforado, Anna Helena Reali Costa

rag fine-tuning benchmarks

arXiv AI

Jun 12

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

arXiv:2606. 13647v1 Announce Type: cross Abstract: We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak.

By Marek \v{S}uppa, Andrej Ridzik, Daniel Hl\'adek, Nat\'alia K\v{n}a\v{z}ekov\'a, Vikt\'oria Ondrejov\'a

rag fine-tuning benchmarks

arXiv AI

Jul 9

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

arXiv:2607. 07669v1 Announce Type: cross Abstract: Large language models increasingly \emph{understand} dialectal English, yet still \emph{produce} only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed.

By Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi, Lu Yin

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 9

MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese

arXiv:2607. 04581v2 Announce Type: replace-cross Abstract: Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated.

By Tardelli Ronan Coelho Stekel

rag benchmarks

arXiv AI

Aug 7

Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation

arXiv:2608. 05785v1 Announce Type: cross Abstract: Multilingual text embedding models are commonly adapted using a single training objective across diverse tasks, despite different tasks requiring fundamentally different optimization strategies.

By Tirth Bhatt, Naren Kumar S, Mayank Singh

rag diffusion benchmarks

arXiv Computation and Language

Sep 1

Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study

arXiv:2608.28776v1 Announce Type: new Abstract: Multilingual sentence embeddings are increasingly used to estimate semantic similarity across languages, yet their sensitivity to fine-grained translat...

By Eleftherios Kalogeros, Athanasios Ntalakas, Manolis Gergatsoulis, Paschalis Nikolaou, Sotiria-Lito Alexaki

rag

arXiv Machine Learning

Jul 7

MTEB-PT: A Text Embedding Benchmark for Brazilian Portuguese

arXiv:2607. 04581v1 Announce Type: cross Abstract: Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated.

By Tardelli Ronan Coelho Stekel

rag benchmarks

arXiv Machine Learning

Jul 7

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

arXiv:2512. 20757v2 Announce Type: replace-cross Abstract: Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs).

By G\"ul Sena Alt{\i}nta\c{s}, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu, Wanru Zhao, Marco Ciccone, Colin Raffel

llms nlp benchmarks

arXiv Machine Learning

Jul 22

Translation as Augmentation: Effect of Translated Data on Assessment of Difficulty

arXiv:2607. 19101v1 Announce Type: cross Abstract: Reliable Text Difficulty Assessment is a prerequisite for valid text simplification workflows and personalized learning applications.

By Yiheng Wu, Jue Hou, Roman Yangarber

nlp

arXiv AI

Jul 24

Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit

arXiv:2512. 10092v2 Announce Type: replace Abstract: Analyzing large-scale text corpora is a core challenge in machine learning, crucial for tasks like identifying undesirable model behaviors or biases in training data.

By Nick Jiang, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, Neel Nanda

llms rag

arXiv AI

Jun 9

Multilingual Training and Evaluation Resources for Vision-Language Models

arXiv:2604. 18347v2 Announce Type: replace-cross Abstract: Vision Language Models (VLMs) achieved rapid progress in the recent years.

By Daniela Baiamonte, Elena Fano, Matteo Gabburo, Stefano Simonazzi, Leonardo Rigutini, Andrea Zugarini

llms multimodal benchmarks

Related stories