arXiv Computation and Language

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

arXiv AI
Aug 12

A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language

arXiv:2608. 10588v1 Announce Type: cross Abstract: Purpose: Fine-grained handshape recognition supports computational sign-language transcription, recognition, and translation, but broad, phonetically defined visual inventories with signer-aware evaluation remain limited.

By Ushnish Sarkar, Suvajit Patra, Bhaswar Chattopadhyay, Pranab Singha Roy, Tapas Samanta
arXiv Computer Vision
Aug 27

Recognising BSL Fingerspelling in Continuous Signing Sequences

The paper introduces FS23K, a large-scale British Sign Language fingerspelling dataset created through an iterative annotation framework. It also presents a recognition model that incorporates bi‑manual interactions and mouthing cues, achieving a halved character error rate compared to previous state‑of‑the‑art methods. These results underscore the dataset’s and model’s value for advancing sign language research and automated annotation pipelines.

By Alyssa Chan, Taein Kwon, Andrew Zisserman
arXiv Computer Vision
Oct 2

Machine Translation for Sign Languages

arXiv:2610.00881v1 Announce Type: new Abstract: Sign language machine translation has progressed substantially over the past decade, evolving from isolated sign recognition to end-to-end translation...

By Ozge Mercanoglu Sincan, Anton Pelykh, Edward Fish, Harry Walsh, JianHe Low, Karahan Sahin, Oline Ranum, Sobhan Asasi, Steven Emery, Richard Bowden
arXiv Computation and Language
Sep 3

SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation

SignBind-LLM introduces a modular framework for sign language translation that separates continuous signing, fingerspelling, and lipreading into dedicated expert streams. Each expert is pre‑trained independently on about two million pseudo‑gloss sequences, eliminating the need for manual gloss annotation. A lightweight transformer fuses the expert outputs, and a pre‑trained language model converts the fused pseudo‑glosses into fluent English, achieving state‑of‑the‑art performance on multiple benchmarks with lower training cost.

By Marshall Thomas, Edward Fish, Richard Bowden
arXiv Computer Vision
Sep 4

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

SignSeek is a new method for learning transferable sign representations that enables efficient retrieval of signs from dictionaries using only a query video. It employs contrastive learning with saliency‑guided articulator masking, aligning same‑gloss signs across signers while focusing on the single most critical articulator per sign. Trained on 266K samples from multiple sign languages, SignSeek achieves state‑of‑the‑art cross‑corpus retrieval performance and zero‑shot generalisation to unseen British Sign Language, also improving isolated sign recognition and subtitle alignment.

By Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden
arXiv Computer Vision
Sep 21

SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

SignGPT is a unified, pose‑based framework that performs gloss‑free sign language translation (SLT) and generation (SLG) by integrating part‑aware hierarchical representations of body, hand, and facial motion into a shared language model. It uses asymmetric multi‑token prediction and progressive training for bidirectional modeling, and is evaluated on How2Sign (ASL) and Phoenix‑2014T (DGS) with benchmark comparisons, qualitative analyses, and component ablations. An exploratory study with 12 Deaf ASL signers demonstrates a sign‑to‑sign response pipeline, suggesting that unified modeling can support sign language conversation (SLC).

By Ronghui Li, Jun Dong, Zhongyuan Hu, Zunnan Xu, Jun Zhou, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, Jie Guo, Xiu Li, Linchao Bao