REVIEW 6 cited by
Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Automatic Sign Language Translation requires the integration of both computer vision and natural language processing to effectively bridge the communication gap between sign and spoken languages. However, the deficiency in large-scale training data to support sign language translation means we need to leverage resources from spoken language. We introduce, Sign2GPT, a novel framework for sign language translation that utilizes large-scale pretrained vision and language models via lightweight adapters for gloss-free sign language translation. The lightweight adapters are crucial for sign language translation, due to the constraints imposed by limited dataset sizes and the computational requirements when training with long sign videos. We also propose a novel pretraining strategy that directs our encoder to learn sign representations from automatically extracted pseudo-glosses without requiring gloss order information or annotations. We evaluate our approach on two public benchmark sign language translation datasets, namely RWTH-PHOENIX-Weather 2014T and CSL-Daily, and improve on state-of-the-art gloss-free translation performance with a significant margin.
Forward citations
Cited by 6 Pith papers
-
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
Hard negatives selected by visual confusability in sign embeddings, not linguistic similarity, substantially raise fine-grained sign-language retrieval accuracy without collapsing coarse performance.
-
Attention-Steered Vision-Language Models for Sign Language Translation
AttnSign adds spatial attention supervision and motion-cadence reinforcement learning to a VLM, improving sign language translation accuracy on How2Sign and OpenASL.
-
Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
LLM-generated pseudo glosses, reordered via weak video supervision, enable sign language translation that rivals gloss-supervised models while needing only 30 gloss examples.
-
Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios
A prompting method that links signs to short text descriptions lets large language models translate English and Italian into sign language glosses, beating prior models in low-data settings.
-
Using Sign Language Production as Data Augmentation to enhance Sign Language Translation
Adding synthetic sign-language data produced by stitching, a GAN, or Gaussian splatting to the training set improves sign-language translation, with the largest gains for skeleton-pose models.
-
ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation
Fusing spatio-temporal RGB and OpenPose features via intra- and cross-modal temporal modeling plus contrastive LLM fine-tuning yields new gloss-free SOTA on PHOENIX14T (BLEU-4 27.10) and CSL-Daily (BLEU-4 25.85).
Discussion (0). Continue with ORCID to comment.