Pith. sign in

REVIEW 4 major objections 8 minor 66 references

SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction

T0 review · 4 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SHuBERT adapts masked cluster prediction to sign language video and claims state-of-the-art transfer results on translation, isolated sign recognition, and fingerspelling detection.

desk verdict A solid multi-stream SSL recipe for ASL that deserves a serious referee; the headline SOTA claims rest on a data-disjointness assertion that is only partially audited. read the letter →

arxiv 2411.16765 v3 pith:ICKIWZLC submitted 2024-11-25 cs.CL cs.CV

Shester Gueuwou , Xiaodan Du , Greg Shakhnarovich , Karen Livescu , Alexander H. Liu This is my paper · ORCID
classification cs.CLcs.CV
keywords signlanguageprocessingself-supervisedlearningmaskedpredictionmulti-streamrepresentationAmericantranslationisolatedrecognitionfingerspellingdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sign language processing has mostly been a collection of task-specific models, each trained from scratch on small labeled datasets. This paper tries to change that by importing the masked-prediction recipe that worked for text and speech into the visual, multi-channel structure of sign language. The result, SHuBERT, is pre-trained on about 984 hours of unlabeled American Sign Language video: it masks parts of hand, face, and body-pose feature streams and learns to predict cluster assignments for the masked parts. A single base model then adapts to translation, isolated sign recognition, and fingerspelling detection, improving over prior systems trained with public data on all but one reported benchmark. If the approach is right, it gives sign language AI a shared foundation model that can grow with unlabeled video, which matters because annotated sign language data is scarce.

What carries the argument

The load-bearing mechanism is masked multi-stream cluster prediction. For each video frame, SHuBERT receives feature vectors for four streams — right hand, left hand, face, and upper-body pose — linearly projected into a joint 1024-dimensional input. K-means clustering, run separately per stream on 10% of the pre-training data, turns each frame's features into discrete pseudo-labels from one of 256 clusters; the transformer is trained only to predict the cluster ids of masked frames, using cross-entropy, so the representation must integrate information across time and across the four channels. The masking span is set to about three frames, roughly 200 milliseconds, approximately the length of a single fingerspelled letter, which the authors identify as the smallest gesture unit. This mechanism is what makes unlabeled video usable: the cluster labels are computed offline, so the model never needs translations or sign-level annotations.

What would settle it

Match every test video in How2Sign, ASL Citizen, Sem-Lex, and WLASL2000 against the pre-training corpus using video-level fingerprinting; if any test clip appears in pre-training, the claimed gains over prior work would be inflated in proportion to the overlap.

Watch

Extended reading notes

Core claim

SHuBERT is a 12-block transformer encoder with 86 million parameters that maps each video frame to a contextual vector given four input streams: left-hand, right-hand, and face appearance features extracted from crops, plus a coarse upper-body pose vector. During pre-training, k-means assigns each frame's features in each stream to one of 256 clusters, and the model is trained to predict the cluster ids of masked spans from the surrounding context; the chosen masking strategy is random span masking with a length of about three frames, matching roughly the duration of a fingerspelled letter. The central discovery is that this objective, applied to roughly 984 hours of public ASL video, yields representations that transfer: after fine-tuning, SHuBERT reports state-of-the-art public-data results on How2Sign and OpenASL translation, on the ASL Citizen and Sem-Lex isolated-sign benchmarks, and on ASL-Stem Wiki fingerspelling detection, with the WLASL2000 benchmark as the single reported exception. The authors also show through ablations that all four streams contribute, that more pre-training data helps, and that a learned weighted sum of all transformer layers is nearly as good as fine-tuning.

Load-bearing premise

The paper's numbers assume that the YouTube-ASL-based pre-training corpus contains none of the test clips from How2Sign, ASL Citizen, Sem-Lex, or WLASL2000, because only the OpenASL overlap is explicitly removed.

Editorial extensions

If this is right

  • A single base SHuBERT model can be adapted to sign language translation, isolated sign recognition, and fingerspelling detection, replacing the previous practice of training a separate specialized model for each task.
  • Pre-training on more unlabeled data improves downstream quality: cutting the pre-training corpus to 10% drops How2Sign translation BLEU from 2.2 to 0.7 in the 100K-step ablation, and the paper expects gains from larger public corpora.
  • Frozen SHuBERT with a learned weighted sum of all layers nearly matches fine-tuned performance, suggesting low-resource tasks can use the pretrained representations without parallel fine-tuning data.
  • Because the cluster targets are computed separately per channel, the same method can be extended to other sign languages by replacing only the pre-training corpus, as the paper states is possible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the four streams and the k-means targets are agnostic to language, the same pre-training pipeline should transfer to other sign languages with equivalent channels; the paper only evaluates American Sign Language.
  • The face stream is deliberately privacy-reduced, greyed out except for eyes and mouth and then blurred, which suggests a path to building larger public pre-training corpora without preserving identifiable faces; the paper does not measure how much accuracy this costs.
  • A direct way to test whether the learned units capture phonological structure would be to feed SHuBERT features into a gloss-based continuous sign recognition model or a sign-language generator; neither is evaluated here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces SHuBERT, a transformer encoder pre-trained on approximately 984 hours of American Sign Language video with a multi-stream masked prediction objective. MediaPipe and DINOv2 features are extracted for the face, left/right hands, and upper-body pose; k-means cluster assignments on these streams serve as pseudo-labels, and masked spans are predicted by four linear classifiers. The resulting contextual representations are fine-tuned for sign language translation (How2Sign, OpenASL, FLEURS-ASL zero-shot), isolated sign recognition (ASL Citizen, Sem-Lex, WLASL2000), and fingerspelling detection (ASL-Stem Wiki), with ablations over masking, data scale, layer weighting, and fine-tuning. The central claim is that a single publicly pre-trained SSL model improves over task-specific public-data baselines on nearly all benchmarks.

Significance. If the empirical claims hold, this is a meaningful step: SHuBERT is one of the first contextual, multi-stream self-supervised representations for sign language, and it is trained entirely on publicly available data. The paper's strengths include explicit separation from private-data systems, a systematic set of ablations, qualitative cluster inspection, and the authors' caution about OpenASL and MSASL overlap. The main risks are data hygiene and statistical reliability; these are addressable and do not invalidate the approach.

major comments (4)
  1. [Sec. 2.2 and Sec. 4.1] The claim that SHuBERT 'is pre-trained on data that is disjoint from the fine-tuning data for the downstream tasks' is verified only for OpenASL (clips are removed) and MSASL (the benchmark is declined because 38.2% of its test set is in YouTube-ASL). No overlap audit is reported for How2Sign, WLASL2000, ASL Citizen, Sem-Lex, or ASL-Stem Wiki. Since YouTube-ASL and YouTube-SL-25 are large web scrapes, and WLASL2000 is built from YouTube ASL videos like MSASL, overlap with these benchmarks is plausible. If any downstream test videos appear in the pre-training corpus, the reported state-of-the-art results in Tables 1-3 would be inflated. Please provide an exhaustive URL/video overlap check between the final pre-training corpus and every evaluated test split, and either exclude overlapping content or report results with and without those clips.
  2. [Sec. 4.1 (pre-training data construction)] The replacement path for OpenASL overlap is incomplete. The paper states that clips from YouTube-ASL that intersect OpenASL are removed and replaced with videos from YouTube-SL-25 that are not present in YouTube-ASL. However, the replacement clips are not also checked against OpenASL. Because footnote 4 says only 72.4% of the OpenASL test set is in YouTube-ASL, the remaining 27.6% of OpenASL test videos could enter the corpus through YouTube-SL-25. This would directly contaminate the OpenASL translation result in Table 1 (+2.0 BLEU). The filtering step should be applied against all downstream benchmark splits, not only against YouTube-ASL.
  3. [Appendix A, Table 4] The masking strategy is selected using a metric whose ranking contradicts the other reported metric. BLEU ranks channel masking highest (2.6), then time (2.3), then random (2.2), whereas BLEURT ranks random highest (31.4), then time (31.2), then channel (29.9). The authors choose random masking because BLEURT aligns better with human judgment, but no confidence intervals or multiple runs are given, and no evidence is provided that this preference transfers to sign language translation. Since masking is a core pre-training design choice, this selection needs either statistical support or a more explicit robustness discussion; otherwise the final configuration is chosen on a metric-dependent basis.
  4. [Tables 1-3] All headline results are point estimates without error bars, confidence intervals, or number of seeds. Several margins over prior work are small: +0.7 BLEU on How2Sign, +0.05 Rec@1 and +0.04 Rec@10 on ASL Citizen, and +0.12 IoU on ASL-Stem-Wiki. Without run-to-run variance, it is not possible to tell whether these differences are significant. Please report mean and standard deviation over multiple seeds for the main comparisons, or at least for the ISLR and fingerspelling experiments where the margins are narrowest.
minor comments (8)
  1. [Sec. 3.1, footnote 3] The claimed ~95% hand detection accuracy on OpenASL is based on 100 videos; please report the exact number of frames or videos and a confidence interval, since this motivates the interpolation step.
  2. [Sec. 2.3] The text contains the typo 'incudes' in the first sentence; it should be 'includes'.
  3. [Sec. 3.2] The word 'unmaksed' appears in the training objective paragraph; it should be 'unmasked'.
  4. [Appendix A] The phrase 'We chose to priotize BLEURT' has a typo: 'priotize' should be 'prioritize'.
  5. [Table 1] The model abbreviations 'V AP' and 'SM' in Table 1 are not expanded in the caption or the main text; please define them.
  6. [Figure 1] The caption says SHuBERT improves 'on all but one task-specific SOTA model'; it would be clearer to name the one exception (WLASL2000, per Sec. 4.3) directly in the caption.
  7. [Sec. 4.2] The text contains the typo 'slighty' in the comparison with SSV-SLT; it should be 'slightly'.
  8. [Sec. 4.3] The sentence 'Uni-Sign fine-tunes 3,000 times more paramters than ours' contains a typo: 'paramters' should be 'parameters'.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation chain; SHuBERT's k-means pseudo-labels are a standard HuBERT-style self-supervised input and the headline results are external-benchmark evaluations, so the main concern (unverified train/test disjointness) is a data-leakage risk rather than circularity.

full rationale

SHuBERT's pre-training targets are k-means cluster assignments computed from the same video-derived features (face, hands, body pose) that the model is trained to reconstruct under masking. This follows the HuBERT recipe and is not circular: the cluster pseudo-labels are an unsupervised training target, not a fitted parameter later renamed as a prediction. The paper's performance claims are evaluated on held-out benchmarks (How2Sign, OpenASL, FLEURS-ASL, ASL Citizen, Sem-Lex, WLASL2000, ASL-Stem Wiki), including a zero-shot FLEURS-ASL setting with no fine-tuning, so the reported gains are not forced by the pre-training objective by construction. The self-citations (e.g., Gueuwou et al. 2025 for multi-stream translation; Sandoval-Castaneda et al. 2023 for SSL ISLR) appear as related-work context and are not load-bearing; no uniqueness theorem or prior claim by the same authors is invoked to forbid alternatives. The most serious validity concern is empirical, not circular: Sec. 2.2 asserts that SHuBERT is 'pre-trained on data that is disjoint from the fine-tuning data for the downstream tasks,' but Sec. 4.1 only describes excluding clips that intersect OpenASL and replacing them from YouTube-SL-25, with no reported overlap audit for How2Sign, ASL Citizen, Sem-Lex, WLASL2000, or ASL-Stem Wiki. If those corpora overlap the pre-training videos, the SOTA improvements could be inflated by leakage; however, this is not a reduction of the derivation to its own inputs and therefore does not constitute circularity. Likewise, choosing the random-masking strategy by BLEURT on How2Sign (Appendix A) is a test-selection concern, not a definitional circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical; the main free choices are the cluster count, masking span, and training schedule. The most consequential assumption is data disjointness, which is only partially verified. No new physical or linguistic entities are introduced.

free parameters (4)
  • k (number of clusters per stream) = 256 per stream
    K-means is computed on 10% of pre-training data; 256 is chosen by hand for each of the four streams and serves as the prediction vocabulary.
  • masking span length = 3 frames (~200 ms)
    Chosen from the average duration of fingerspelling a single ASL letter, cited to Hanson (1982).
  • pre-training schedule = 400K steps, peak LR 5e-4, 8% warmup, 8 A6000 GPUs
    Hyperparameters selected for training; results are sensitive to training length as shown by 100K-step ablations.
  • DINOv2 continued pre-training = 5 million crops per stream, 1 epoch
    The feature extractor is further trained on 5M face and 5M hand crops from YouTube-ASL; the effect of this step is not ablated.
assumptions (5)
  • domain assumption The k-means cluster assignments on DINOv2 features are learnable targets that encode linguistic distinctions.
    Required for the masked cluster prediction objective to produce useful representations. The paper supports this with qualitative cluster visualizations and downstream gains, but it is not proven.
  • domain assumption Pre-training data is disjoint from downstream fine-tuning data for all evaluated benchmarks.
    Explicitly validated only for OpenASL and MSASL; How2Sign, ASL Citizen, Sem-Lex, and WLASL2000 overlaps are not checked (Sec. 4.1).
  • domain assumption MediaPipe landmark detection provides sufficiently accurate hand, face, and body pose inputs on natural ASL video.
    The paper estimates about 95% hand detection accuracy on 100 OpenASL videos and interpolates failures; downstream performance relies on this preprocessing.
  • standard math BLEURT is a more reliable translation metric than BLEU for model selection.
    Used to justify choosing random masking over channel masking despite BLEU favoring channel masking (Appendix A, citing Freitag et al. 2022).
  • domain assumption DINOv2 features, after continued pre-training, cluster semantically for sign language handshapes and facial expressions.
    Relied on for the quality of cluster targets; the paper notes visual inspection suggests clusters are gesture-related rather than appearance-related.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction." pith.science (2026). https://pith.science/paper/ICKIWZLC

@misc{pith2026241116765,
  author       = {Pith},
  title        = {Pith review of: SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ICKIWZLC}},
  note         = {Machine review of arXiv:2411.16765}
}
read the original abstract

Sign language processing has traditionally relied on task-specific models, limiting the potential for transfer learning across tasks. Pre-training methods for sign language have typically focused on either supervised pre-training, which cannot take advantage of unlabeled data, or context-independent (frame or video segment) representations, which ignore the effects of relationships across time in sign language. We introduce SHuBERT (Sign Hidden-Unit BERT), a self-supervised contextual representation model learned from approximately 1,000 hours of American Sign Language video. SHuBERT adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams. SHuBERT achieves state-of-the-art performance across multiple tasks including sign language translation, isolated sign language recognition, and fingerspelling detection.

Figures

Figures reproduced from arXiv: 2411.16765 by the authors.

Figure 1
Figure 1. Comparison between our results using fine [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SHuBERT pre-training. (a) We locate a set of landmarks in each frame of the input video using [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Sample frames of several signers and the cor [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Three strategies for sequence masking. Each [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Sample face clusters. Each row represents a distinct cluster and 10 random examples from it. Cluster 141 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Sample left hand clusters. Each row represents a distinct cluster and 10 random examples from it. Cluster [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Sample right hand clusters. Each row represents a distinct cluster and 10 random examples from it. Cluster [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Sample upper body clusters. Each row represents a distinct cluster and 10 random examples from it. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 53 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Bbc-oxford british sign language dataset

    Samuel Albanie, G \"u l Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al. Bbc-oxford british sign language dataset. arXiv preprint arXiv:2111.03635, 2021

    arXiv 2021

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, 2020

    work page 2020

  4. [4]

    A comparison of sign language and spoken language

    Ursula Bellugi and Susan Fischer. A comparison of sign language and spoken language. Cognition, 1972

    work page 1972

  5. [5]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  6. [6]

    Sign language recognition, generation, and translation: An interdisciplinary perspective

    Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, et al. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, 2019

    work page 2019

  7. [7]

    Neural sign language translation

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018

    work page 2018

  8. [8]

    Multi-channel transformers for multi-articulatory sign language translation

    Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Multi-channel transformers for multi-articulatory sign language translation. In Computer Vision--ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part IV 16, 2020

    work page 2020

Show all 66 references
  1. [9]

    A simple multi-modality transfer learning baseline for sign language translation

    Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022 a

    2022

  2. [10]

    Two-stream network for sign language recognition and translation

    Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, 2022 b

    2022

  3. [11]

    C ^2 rl: Content and context representation learning for gloss-free sign language translation and retrieval, 2024

    Zhigang Chen, Benjia Zhou, Yiqing Huang, Jun Wan, Yibo Hu, Hailin Shi, Yanyan Liang, Zhen Lei, and Du Zhang. C ^2 rl: Content and context representation learning for gloss-free sign language translation and retrieval, 2024

    2024

  4. [12]

    Asl citizen: a community-sourced dataset for advancing isolated sign language recognition

    Aashaka Desai, Lauren Berger, Fyodor Minakov, Nessa Milano, Chinmay Singh, Kriston Pumphrey, Richard Ladner, Hal Daum \'e III, Alex X Lu, Naomi Caselli, et al. Asl citizen: a community-sourced dataset for advancing isolated sign language recognition. In Advances in Neural Info...

    2024

  5. [13]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lan...

    2019

  6. [14]

    How2sign: a large-scale multimodal dataset for continuous american sign language

    Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2sign: a large-scale multimodal dataset for continuous american sign language. In Proceedings of the IEEE/CVF Conference on Computer Visio...

    2021

  7. [15]

    Fingerspelling posenet: Enhancing fingerspelling translation with pose-based transformer models

    Pooya Fayyazsanavi, Negar Nejatishahidin, and Jana Ko s eck \'a . Fingerspelling posenet: Enhancing fingerspelling translation with pose-based transformer models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024

    2024

  8. [16]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics are better and more robust. In Proceedings o...

    2022

  9. [17]

    Translationese in swedish novels translated from english

    Martin Gellerstam. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 1986

    1986

  10. [18]

    Fsboard: Over 3 million characters of asl fingerspelling collected via smartphones

    Manfred Georg, Garrett Tanzer, Saad Hassan, Maximus Shengelia, Esha Uboweja, Sam Sepah, Sean Forbes, and Thad Starner. Fsboard: Over 3 million characters of asl fingerspelling collected via smartphones. arXiv preprint arXiv:2407.15806, 2024

    2024 arXiv

  11. [19]

    JWS ign: A highly multilingual corpus of B ible translations for more diversity in sign language processing

    Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias M \"u ller. JWS ign: A highly multilingual corpus of B ible translations for more diversity in sign language processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023

    2023

  12. [20]

    Signmusketeers: An efficient multi-stream approach for sign language translation at scale

    Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. Signmusketeers: An efficient multi-stream approach for sign language translation at scale. arXiv preprint arXiv:2406.06907, 2024

    2024 arXiv

  13. [21]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021

    2021

  14. [22]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  15. [23]

    Signbert: Pre-training of hand-model-aware representation for sign language recognition

    Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, and Houqiang Li. Signbert: Pre-training of hand-model-aware representation for sign language recognition. In Proceedings of the IEEE/CVF international conference on computer vision, 2021

    2021

  16. [24]

    Sign BERT +: Hand-model-aware self-supervised pre-training for sign language understanding

    Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. Sign BERT +: Hand-model-aware self-supervised pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  17. [25]

    Skeleton aware multi-modal sign language recognition

    Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021

    2021

  18. [26]

    Signclip: Connecting text and sign language by contrastive learning

    Zifan Jiang, Gerard Sant, Amit Moryossef, Mathias M \"u ller, Rico Sennrich, and Sarah Ebling. Signclip: Connecting text and sign language by contrastive learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  19. [27]

    The sem-lex benchmark: Modeling asl signs and their phonemes

    Lee Kezar, Jesse Thomason, Naomi Caselli, Zed Sehyr, and Elana Pontecorvo. The sem-lex benchmark: Modeling asl signs and their phonemes. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, 2023 a

    2023

  20. [28]

    Improving sign recognition with phonology

    Lee Kezar, Jesse Thomason, and Zed Sehyr. Improving sign recognition with phonology. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023 b

    2023

  21. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, 2014

    2014

  22. [30]

    Translationese and its dialects

    Moshe Koppel and Noam Ordan. Translationese and its dialects. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, 2011

    2011

  23. [31]

    Testing mediapipe holistic for linguistic analysis of nonmanual markers in sign languages

    Anna Kuznetsova and Vadim Kimmelman. Testing mediapipe holistic for linguistic analysis of nonmanual markers in sign languages. arXiv preprint arXiv:2403.10367, 2024

    2024 arXiv

  24. [32]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020

    2020

  25. [33]

    Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison

    Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, 2020

    2020

  26. [34]

    Gloss-free end-to-end sign language translation

    Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Bang Zhang, and Yi Yang. Gloss-free end-to-end sign language translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023

    2023

  27. [35]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  28. [36]

    Mediapipe: A framework for building perception pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019

    1906 arXiv

  29. [37]

    Evaluating the immediate applicability of pose estimation for sign language recognition

    Amit Moryossef, Ioannis Tsochantaridis, Joe Dinn, Necati Cihan Camgoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling. Evaluating the immediate applicability of pose estimation for sign language recognition. In Proceedings of the IEEE/CVF conference ...

    2021

  30. [38]

    Why alternative gloss labels will increase the value of the wlasl dataset

    Carol Neidle and Carey Ballard. Why alternative gloss labels will increase the value of the wlasl dataset. In webiste, 2022

    2022

  31. [39]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2023

    2023

  32. [40]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002

    2002

  33. [41]

    A call for clarity in reporting BLEU scores

    Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, 2018

    2018

  34. [42]

    Sign language recognition with multi-modal features

    Junfu Pu, Wengang Zhou, and Houqiang Li. Sign language recognition with multi-modal features. In Advances in Multimedia Information Processing-PCM 2016: 17th Pacific-Rim Conference on Multimedia, Xi \' an, China, September 15-16, 2016, Proceedings, Part II , 2016

    2016

  35. [43]

    Improving language understanding by generative pre-training, 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018

    2018

  36. [44]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 2020

    2020

  37. [45]

    Towards privacy-aware sign language translation at scale

    Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, and Jean Maillard. Towards privacy-aware sign language translation at scale. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024

    2024

  38. [46]

    Self-supervised video transformers for isolated sign language recognition

    Marcelo Sandoval-Castaneda, Yanhong Li, Diane Brentari, Karen Livescu, and Gregory Shakhnarovich. Self-supervised video transformers for isolated sign language recognition. arXiv preprint arXiv:2309.02450, 2023

    2023 arXiv

  39. [47]

    The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language

    Zed Sevcikova Sehyr, Naomi Caselli, Ariel M Cohen-Goldberg, and Karen Emmorey. The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language. In The Journal of Deaf Studies and Deaf Education, 2021

    2021

  40. [48]

    Bleurt: Learning robust metrics for text generation

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2020

    2020

  41. [49]

    Toward American Sign Language processing in the real world: Data, tasks, and methods

    Bowen Shi. Toward American Sign Language processing in the real world: Data, tasks, and methods. arXiv preprint arXiv:2308.12419, 2023

    2023 arXiv

  42. [50]

    Fingerspelling recognition in the wild with iterative visual attention

    Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Fingerspelling recognition in the wild with iterative visual attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019

    2019

  43. [51]

    Open-domain sign language translation learned from online video

    Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022 a

    2022

  44. [52]

    Learning audio-visual speech representation by masked multimodal cluster prediction

    Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. In International Conference on Learning Representations, 2022 b

    2022

  45. [53]

    Fingerspelling within sign language translation, 2024 a

    Garrett Tanzer. Fingerspelling within sign language translation, 2024 a

    2024

  46. [54]

    Fleurs-asl: Including american sign language in massively multilingual multitask evaluation

    Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multitask evaluation. arXiv preprint arXiv:2408.13585, 2024 b

    2024 arXiv

  47. [55]

    Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus

    Garrett Tanzer and Biao Zhang. Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus. arXiv preprint arXiv:2407.11144, 2024

    2024 arXiv

  48. [56]

    Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus

    Dave Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. In Advances in Neural Information Processing Systems, 2023

    2023

  49. [57]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017

    2017

  50. [58]

    Use of orthographic structure by deaf adults: Recognition of fingerspelled words

    Hanson VL. Use of orthographic structure by deaf adults: Recognition of fingerspelled words. In Applied Psycholinguistics, 1982

    1982

  51. [59]

    Lin, Andy T

    Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mo...

    2021

  52. [60]

    Sign2 GPT : Leveraging large language models for gloss-free sign language translation

    Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2 GPT : Leveraging large language models for gloss-free sign language translation. In The Twelfth International Conference on Learning Representations, 2024

    2024

  53. [61]

    Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification

    Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), 2018

    2018

  54. [62]

    B y T 5: Towards a token-free future with pre-trained byte-to-byte models

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. B y T 5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 2022

    2022

  55. [63]

    Including signed languages in natural language processing

    Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on ...

    2021

  56. [64]

    Scaling sign language translation

    Biao Zhang, Garrett Tanzer, and Orhan Firat. Scaling sign language translation. In Advances in Neural Information Processing Systems, 2024

    2024

  57. [65]

    Stylebreeder: Exploring and democratizing artistic styles through text-to-image models

    Matthew Zheng, Enis Simsar, Hidir Yesiltepe, Federico Tombari, Joel Simon, and Pinar Yanardag. Stylebreeder: Exploring and democratizing artistic styles through text-to-image models. In Advances in Neural Information Processing Systems, 2024

    2024

  58. [66]

    Spatial-temporal multi-cue network for sign language recognition and translation

    Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia, 2021

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.