← Back to all news
arXiv AI By Oline Ranum, Edward Fish, Simon Hadfield, Richard Bowden

Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks

Read the original on arXiv AI →

The paper argues that BLEU-4, the prevailing metric for sign language translation (SLT), may not accurately reflect sign language proficiency because SLT models can exploit spurious correlations and spoken-language priors. By evaluating six SLT models on Phoenix-2014T and CSL-Daily, the authors show that higher BLEU-4 scores do not necessarily indicate better spatio-temporal understanding. They propose a new open-weight LLM QA protocol inspired by language-learning assessment, which better preserves salient content, aligns more closely with human rankings, and reveals differences between gloss-free and gloss-supervised systems that BLEU-4 obscures.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 9

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

SignRAG is a unified framework for gloss‑free sign language translation that combines hierarchical pretraining, target‑domain retrieval augmentation, and retrieval‑aware reinforcement fine‑tuning. It first learns linguistically grounded sign representations, aligns a sign encoder with a decoder‑only large language model, and then uses a retrieval gallery to provide instance‑specific translation cues. Retrieval Utility‑Guided Reinforcement Fine‑Tuning (RUG‑RFT) balances translation quality and retrieval usefulness, leading to state‑of‑the‑art results, including the first gloss‑free approach to outperform gloss‑supervised methods on CSL‑Daily.

By Zhi Rao, Yucheng Zhou, Qianran Sun, Yiqing Huang, Longcan Yuan, Jiayi Hou, Chengwen Yao, Lin Cheng, Donghui Sun, Xiaoxin Chen, Jun Wan
llmsragfine-tuningmultimodalbenchmarks
More like this →
arXiv Computation and Language
Oct 1

Direct Translation between Sign Languages

arXiv:2605.20588v2 Announce Type: replace Abstract: Sign language translation has made substantial progress between sign and spoken languages, while translation across sign languages remains less exp...

By Zetian Wu, Bowen Xie, Wuyang Meng, Milan Gautam, Stefan Lee, Liang Huang
More like this →
arXiv Computer Vision
Sep 4

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

SignSeek is a new method for learning transferable sign representations that enables efficient retrieval of signs from dictionaries using only a query video. It employs contrastive learning with saliency‑guided articulator masking, aligning same‑gloss signs across signers while focusing on the single most critical articulator per sign. Trained on 266K samples from multiple sign languages, SignSeek achieves state‑of‑the‑art cross‑corpus retrieval performance and zero‑shot generalisation to unseen British Sign Language, also improving isolated sign recognition and subtitle alignment.

By Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden
ragdiffusionfine-tuningbenchmarkssafety
More like this →
arXiv AI
Jun 19

Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models

arXiv:2605. 31393v2 Announce Type: replace-cross Abstract: Sign language translation (SLT) remains constrained by the limited availability of paired sign-video/text corpora and by the heavy-tailed vocabularies typical of real-world datasets.

By Pedro Dal Bianco, Jean Paul Nunes Reinhold, Oscar Stanchi, Facundo Quiroga, Franco Ronchetti, Ulisses Brisolara Corr\^ea
llmsfine-tuningefficiency
More like this →