A prompting method that links signs to short text descriptions lets large language models translate English and Italian into sign language glosses, beating prior models in low-data settings.
SignCLIP: Connecting Text and Sign Language by Contrastive Learning
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We present SignCLIP, which re-purposes CLIP (Contrastive Language-Image Pretraining) to project spoken language text and sign language videos, two classes of natural languages of distinct modalities, into the same space. SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs, without directly optimizing for a specific task or sign language which is often of limited size. We pretrain SignCLIP on Spreadthesign, a prominent sign language dictionary consisting of ~500 thousand video clips in up to 44 sign languages, and evaluate it with various downstream datasets. SignCLIP discerns in-domain signing with notable text-to-video/video-to-text retrieval accuracy. It also performs competitively for out-of-domain downstream tasks such as isolated sign language recognition upon essential few-shot prompting or fine-tuning. We analyze the latent space formed by the spoken language text and sign language poses, which provides additional linguistic insights. Our code and models are openly available.
fields
cs.CL 1years
2025 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios
A prompting method that links signs to short text descriptions lets large language models translate English and Italian into sign language glosses, beating prior models in low-data settings.