arXiv AI September 17, 2026 By Martin Kosteln\'ik, Michal Hradi\v{s}, Martin Do\v{c}ekal

CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents

The paper introduces CzechTopic, a human‑annotated benchmark for zero‑shot topic localization in historical Czech documents. It provides topics with manually annotated spans and evaluates models at both document and word levels, using human agreement as the reference. Experiments show wide performance differences among large language models, with the best models approaching human agreement while smaller distilled token‑embedding models remain competitive.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 12

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

arXiv:2606. 13647v1 Announce Type: cross Abstract: We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak.

By Marek \v{S}uppa, Andrej Ridzik, Daniel Hl\'adek, Nat\'alia K\v{n}a\v{z}ekov\'a, Vikt\'oria Ondrejov\'a

rag fine-tuning benchmarks

arXiv Computation and Language

Oct 9

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).

By Hala Almaghout, Christian Federmann, Qin Gao

llms nlp

arXiv Computation and Language

Sep 22

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

arXiv:2605.28190v2 Announce Type: replace Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...

By Manuel Frank, Haithem Afli

llms rag benchmarks

arXiv Computation and Language

Sep 2

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

arXiv:2607.00171v2 Announce Type: replace Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover on...

By Andrianos Michail, Stylianos Psychias, Michelle Wastl, Simon Clematide, Rico Sennrich, Juri Opitz

rag nlp benchmarks

arXiv AI

Jul 1

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

arXiv:2606. 31718v1 Announce Type: cross Abstract: Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora.

By Dragos-Mitrut Vasile, Elena-Simona Apostol, Stefan-Adrian Toma, Adrian Paschke, Ciprian-Octavian Truica

llms fine-tuning benchmarks

arXiv Computation and Language

Sep 30

RunyaNER: Auxiliary Language Selection for Runyankore NER

arXiv:2609.37543v1 Announce Type: new Abstract: Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-reso...

By Prosper Arineitwe Asiimwe, Francois Meyer, Jan Buys

rag nlp fine-tuning benchmarks

One email a morning, machine-written

Related stories

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

RunyaNER: Auxiliary Language Selection for Runyankore NER