SignMatch introduces a prototype‑structured embedding space that learns to match dictionary sign videos with continuous sign language footage based solely on visual similarity of handshape and motion. By mapping isolated dictionary exemplars into this space, the method enables direct, embedding‑based sign matching and can generalise to unseen signs using only dictionary examples. Experiments on ASL‑Citizen, ChaLearn OSLWL, and BOBSL CSLR2 benchmarks show strong cross‑dataset, cross‑task, and cross‑language performance, outperforming prior approaches on American, British, and Spanish sign languages without benchmark‑specific supervision.
By Ryan Wong, Youngjoon Jang, Liliane Momeni, G\"ul Varol, Andrew Zisserman
SignRAG is a unified framework for gloss‑free sign language translation that combines hierarchical pretraining, target‑domain retrieval augmentation, and retrieval‑aware reinforcement fine‑tuning. It first learns linguistically grounded sign representations, aligns a sign encoder with a decoder‑only large language model, and then uses a retrieval gallery to provide instance‑specific translation cues. Retrieval Utility‑Guided Reinforcement Fine‑Tuning (RUG‑RFT) balances translation quality and retrieval usefulness, leading to state‑of‑the‑art results, including the first gloss‑free approach to outperform gloss‑supervised methods on CSL‑Daily.
By Zhi Rao, Yucheng Zhou, Qianran Sun, Yiqing Huang, Longcan Yuan, Jiayi Hou, Chengwen Yao, Lin Cheng, Donghui Sun, Xiaoxin Chen, Jun Wan
The paper introduces a reverse sign language dictionary that recognizes signs from continuous signing without relying on gloss annotations. It does this by captioning sign-level video clips into free-form procedural descriptions using an open-weight vision‑language model, then retrieving the closest description from a multilingual sentence encoder’s vocabulary. Experiments on a Japanese Sign Language corpus show that fine‑tuning the captioner boosts seen‑class retrieval from 4.5% to 49% and improves unseen‑class retrieval from 11.5% to 21%, matching or surpassing traditional closed‑set classifiers where applicable.
arXiv:2608. 06407v1 Announce Type: cross Abstract: Automated Sign Language Recognition for under-represented languages remains a largely unsolved problem.
By Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi, Emmanuel Aaron, Moise Busogi
The paper introduces a reverse sign‑language dictionary that recognizes signs from continuous signing without relying on gloss labels. It does this by first captioning a sign‑level video clip into a free‑form procedural description using an open‑weight vision‑language model, then retrieving the closest entry from a multilingual sentence encoder’s vocabulary of target descriptions. Experiments on a Japanese Sign Language dialogue corpus show that fine‑tuning the captioner boosts seen‑class retrieval from 4.5 % to 49 % and improves unseen‑class retrieval from 11.5 % to 21 %, approaching the performance of a standard closed‑set classifier while enabling open‑vocabulary recognition.
By Santiago Poveda-Guti\'errez, Hideki Nakayama, Mayumi Bono
arXiv:2609.07965v1 Announce Type: cross
Abstract: Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev...
By Keren Artiaga, Sabyasachi Kamila, Haithem Afli, Conor Lynch, Mohammed Hasanuzzaman
arXiv:2608. 09045v1 Announce Type: cross Abstract: Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress.
By Xiao Liu, Shiwei Gan, Yafeng Yin, Jiaxin Yin, Bowen Guo, Yaqi Sun, Zhiwei Jiang, Lei Xie
arXiv:2609.25862v1 Announce Type: new
Abstract: We present the first experiments on isolated sign language recognition (ISLR) for Icelandic Sign Language (\'ITM). We use \'ITM SignWiki, a dataset der...
By Finnur \'Ag\'ust Ingimundarson, Gu{\dh}n\'y Bj\"ork {\TH}orvaldsd\'ottir, Mathias M\"uller, Sarah Ebling
arXiv:2608. 09006v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks.
By Shiwei Gan, Xiao Liu, Yafeng Yin, Zhiwei Jiang, Bowen Guo, Lie Xie, Sanglu Lu, Hongkai Wen
The paper introduces FS23K, a large-scale British Sign Language fingerspelling dataset created through an iterative annotation framework. It also presents a recognition model that incorporates bi‑manual interactions and mouthing cues, achieving a halved character error rate compared to previous state‑of‑the‑art methods. These results underscore the dataset’s and model’s value for advancing sign language research and automated annotation pipelines.
By Alyssa Chan, Taein Kwon, Andrew Zisserman
Isharah‑Selfie is a large‑scale Arabic Sign Language dataset collected entirely with front‑facing smartphone cameras in unconstrained selfie conditions. It contains 8,997 video clips from 16 signers across 1,493 unique sentences, annotated with gloss sequences for continuous sign language recognition and Arabic sentence translations for sign language translation. The dataset focuses on realistic mobile communication, featuring one‑handed signing, close‑range viewpoints, partial body visibility, camera motion, unconstrained environments, and heterogeneous device resolutions, and introduces signer‑independent and unseen‑sentence evaluation protocols.
By Ahmed Abul Hasanaath, Murtadha Aljubran, Sarah Alyami, Muhammad Haris Khan, Hamzah Luqman
SignBind-LLM introduces a modular framework for sign language translation that separates continuous signing, fingerspelling, and lipreading into dedicated expert streams. Each expert is pre‑trained independently on about two million pseudo‑gloss sequences, eliminating the need for manual gloss annotation. A lightweight transformer fuses the expert outputs, and a pre‑trained language model converts the fused pseudo‑glosses into fluent English, achieving state‑of‑the‑art performance on multiple benchmarks with lower training cost.
By Marshall Thomas, Edward Fish, Richard Bowden