← Back to all news
arXiv AI By Martin Kosteln\'ik, Michal Hradi\v{s}, Martin Do\v{c}ekal

CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents

Read the original on arXiv AI →

The paper introduces CzechTopic, a human‑annotated benchmark for zero‑shot topic localization in historical Czech documents. It provides topics with manually annotated spans and evaluates models at both document and word levels, using human agreement as the reference. Experiments show wide performance differences among large language models, with the best models approaching human agreement while smaller distilled token‑embedding models remain competitive.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 12

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

arXiv:2606. 13647v1 Announce Type: cross Abstract: We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak.

By Marek \v{S}uppa, Andrej Ridzik, Daniel Hl\'adek, Nat\'alia K\v{n}a\v{z}ekov\'a, Vikt\'oria Ondrejov\'a
ragfine-tuningbenchmarks
More like this →
arXiv Computation and Language
Sep 2

Does task decomposition improve automatic NLG evaluation?

The paper evaluates whether breaking down evaluation tasks into simpler sub‑tasks improves the LLM-as-a-judge (LLMaJ) framework for reference‑free NLG assessment. Systematic comparisons across multiple datasets show no performance advantage for LLMaJ methods that use task decomposition over a fair baseline that does not. The authors attribute previously reported gains to the use of human labels for training rather than to decomposition itself, and note that LLMaJ without decomposition can match human annotators when such labels are available.

By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
llms
More like this →