Pith. sign in

REVIEW 3 major objections 4 minor 96 references

Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sign language generation becomes autoregressive token prediction.

desk verdict Solid, honest engineering paper with a real efficiency win, but the CSL/Phoenix SOTA numbers should be read with caution because the ground truth and training labels come from the same pose-fitting pipeline. read the letter →

arxiv 2411.17799 v3 pith:NRD5DXIB submitted 2024-11-26 cs.CV cs.CL

Ronglai Zuo , Rolandos Alexandros Potamias , Evangelos Ververas , Jiankang Deng , Stefanos Zafeiriou This is my paper · ORCID
classification cs.CVcs.CL
keywords signlanguagegenerationtext-to-signmultilingualmodeldecoupledtokenizermulti-headdecodingretrieval-enhancedSMPL-XavatarVQ-VAE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that sign language generation—turning a written sentence into a 3D signing avatar—should be framed as a language modeling problem rather than a visual generation problem. It introduces SOKE, a single multilingual generator for American, Chinese, and German sign languages, and claims that the discrete, multi-cue structure of signs can be captured by a decoupled tokenizer that turns continuous motion into separate token streams for the upper body and each hand. A pretrained multilingual language model then generates those tokens autoregressively from text, helped by retrieved word-level dictionary signs. If this claim is right, text-to-sign can be handled by one scalable model that improves with more data and more languages, rather than by a separate model per sign language or per dataset.

What carries the argument

The mechanism that carries the argument is the decoupled tokenizer combined with multi-head decoding and retrieval-conditioned prompting. The decoupled tokenizer discretizes continuous SMPL-X sign motions into three independent token streams (upper body codebook of 96 codes, left-hand and right-hand codebooks of 192 codes each), so the model can represent the simultaneous body and hand cues that carry sign semantics. The multi-head decoder predicts all three part-wise tokens at once at each step, with input embeddings fused as $E=(1-2\lambda)E_B+\lambda E_{LH}+\lambda E_{RH}$; this cuts the decoding steps from $3K$ to $K$ while still sharing context across body parts. The retrieval-enhanced condition prepends the motion tokens of lemmatized dictionary words to the text prompt, giving the encoder precise word-level signs to anchor sentence-level generation.

What would settle it

Fitting the same pipeline on a mocap or mesh-annotated dataset of Chinese and German continuous signs would settle it: if hand or body fitting errors on those languages are as large as the reported DTW differences between SOKE and its closest baseline, the DTW gains measure fit consistency rather than sign correctness. A blind rating by native signers of whether generated signs convey the source text, comparing SOKE with S-MotionGPT on the same examples, would test the linguistic claim directly.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that sign language production can be reduced to next-token prediction over three parallel part-wise token streams. SOKE trains three VQ-VAE codebooks—one for the upper body, one for the left hand, one for the right hand—to discretize SMPL-X sign motions, then inserts these tokens into the vocabulary of a multilingual language model. At each decoding step three heads predict the body and hand tokens simultaneously, with the decoder input at each step set to the weighted average embedding $E = (1-2\lambda)E_B+\lambda E_{LH}+\lambda E_{RH}$ with $\lambda = 1/3$. Word-level dictionary signs, converted to motion tokens, are appended to the text prompt as auxiliary conditions. The paper reports state-of-the-art procrustes-aligned DTW body errors of 2.35, 1.71, and 1.38 on How2Sign, CSL-Daily, and Phoenix-2014T, with back-translation BLEU-4 of 14.48, 11.30, and 11.87, and shows in ablations that multilingual training, the decoupled tokenizer, multi-head decoding, and retrieval each contribute.

Load-bearing premise

The load-bearing premise is that the SMPL-X poses fitted for CSL-Daily and Phoenix-2014T are accurate enough to serve as both training targets and DTW evaluation ground truth; the paper validates them qualitatively on those datasets and quantitatively only on a different benchmark, SGNify.

Editorial extensions

If this is right

  • Text-to-sign generation does not need per-language models: a single fine-tuned multilingual language model can serve multiple sign languages.
  • Because encoding and decoding scale with data, adding more sign languages and larger sign dictionaries should keep improving generation quality and vocabulary coverage.
  • Multi-head decoding cuts inference steps from three times the motion length to the motion length itself, making real-time avatar signing a realistic deployment target.
  • Retrieval of word-level dictionary signs improves the precision of generated signs by around one-fifth on each benchmark without inserting unnatural dictionary poses directly into the output.
  • Separating upper-body, left-hand, and right-hand tokenization consistently outperforms whole-body tokenization, indicating that hand–body decomposition is central to sign semantics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to let native signers rate semantic equivalence of the generated signs directly against the source text; this would separate meaning fidelity from pose-fit consistency.
  • The paper's argument implies that the same tokenizer plus language-model recipe could work in reverse for sign-to-text translation, forming a truly bilingual text–sign model within one vocabulary.
  • Because the retrieval condition uses word-level dictionary signs, the expected gain should depend on dictionary coverage per language; comparing retrieval gains across languages with controlled dictionary sizes would test this dependence.
  • The reported DTW numbers on CSL-Daily and Phoenix-2014T measure agreement with the paper's fitted SMPL-X targets; if independently mocap-annotated data for those languages become available, re-running the comparison would tell whether the gains persist.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SOKE, a unified multilingual sign language generator from text to 3D avatar motion. The method has two stages: a decoupled VQ-VAE tokenizer that discretizes upper-body, left-hand, and right-hand motion into separate token streams, and an autoregressive multilingual generator built on mBART-large, which predicts the three token streams jointly via a multi-head decoding strategy. The generator is conditioned on text and on motion tokens retrieved from external word-level sign dictionaries. The authors evaluate on How2Sign, CSL-Daily, and Phoenix-2014T, reporting state-of-the-art DTW-PA-JPE values of 2.35, 1.71, and 1.38 and back-translation BLEU-4 scores of 14.48, 11.30, and 11.87. Ablations cover decoding strategies, retrieval, multilingual scaling, tokenizer decomposition, and codebook sizes, and a small user study with professional signers is included. For CSL-Daily and Phoenix-2014T, the SMPL-X ground-truth poses are produced by the authors' own OSX/WiLoR/Mediapipe fitting pipeline.

Significance. The paper's core idea—treating sign language production as tokenized autoregressive prediction with a pretrained multilingual LM—is timely and technically coherent. The decoupled tokenizer, the multi-head decoding scheme that reduces decoding steps from O(3K) to O(K), and the retrieval mechanism that uses dictionary motion tokens as conditions rather than splicing them into the output are all reasonable design contributions. The ablations are internally consistent: multi-head decoding improves over both sequential and parallel decoding, retrieval consistently reduces DTW error, and multilingual training helps. If the reported gains are externally valid, SOKE would be a meaningful step toward practical multilingual text-to-sign systems. The main risk is that two of the three benchmark evaluations use labels produced by the same pose-fitting pipeline that generated the training targets, so the DTW numbers on CSL-Daily and Phoenix-2014T may partly reflect consistency with fitted poses rather than linguistically correct signing. The quantitative validation in Table S5 is on a different dataset with a different protocol and therefore does not fully remove this concern.

major comments (3)
  1. [§3.1, Tables 1 and S5, Figure S8] The ground-truth SMPL-X poses for CSL-Daily and Phoenix-2014T are generated by the authors' OSX/WiLoR/Mediapipe fitting pipeline, and the same fitted poses serve simultaneously as training regression targets and as DTW evaluation labels. If the fitting pipeline has systematic dataset-specific errors (e.g., Mediapipe-based upper-body refinement biases or WiLoR hand-parameter substitution artifacts), the model can learn to reproduce those errors during training, and the DTW metric will reward that reproduction even when the produced sign is linguistically wrong. Table S5 validates the pipeline on SGNify, which has different signers, capture setup, and an evaluation metric (mean per-vertex error) different from the DTW protocol used in Table 1; it therefore cannot establish that the CSL-Daily and Phoenix-2014T fitted targets are free of dataset-specific systematic errors. The paper's own Limitations section concedes that reconstruction errors are inevitable, and Figure S8 is only qualitative. This is load-bearing because the central SOTA claim on two of the three benchmarks depends on the accuracy of these fitted labels. Please provide a quantitative validation of the fitted poses on CSL-Daily and Phoenix-2014T (e.g., against manual joint annotations, alternative estimation methods, or signer evaluation), or evaluate the generated signs with at least one protocol that does not use the same fitted poses as ground truth (e.g., an independent sign-language recognizer or human ratings for DGS/CSL).
  2. [§4.1, Tables 1 and 2] None of the quantitative claims in Tables 1 and 2 is accompanied by error bars, bootstrap intervals, or significance tests. The text repeatedly uses 'significantly lower' and 'significantly enhances', but all numbers appear to come from a single run or a single evaluation pass. Given that the comparisons include reimplemented baselines and that the central claim is state-of-the-art performance, the absence of variance estimates makes it impossible to assess whether the reported margins are robust. Please report results over multiple random seeds or bootstrap resampling, and where relevant perform a paired significance test.
  3. [§3.4] The retrieval dictionary is filtered by retaining, for each word, only the instance with the lowest reconstruction error after passing through the tokenizer. Because the tokenizer is trained on the same distribution and the same fitting pipeline as the evaluation data, this selection criterion can systematically favor entries that are easiest for the model to reconstruct, which may inflate the retrieval gains reported in Table 2. The paper does not report dictionary coverage, retrieval frequency, or an ablation in which unfiltered or randomly selected dictionary instances are used. Please provide these analyses to show that the retrieval benefit is not an artifact of the selection rule.
minor comments (4)
  1. [§4.3, Figure 7] The user study involves only 9 professional signers and 15 generated signs per method, and no statistical test is reported for the rating differences. A paired test or confidence intervals would strengthen the claim that the ratings differ beyond noise.
  2. [§2] The typo 'addtional' appears in the Related Work section; please correct it.
  3. [Table S6] The header of Table S6 contains a formatting error ('N B Z N LH Z = N RH Z' is repeated and interspersed with column values), making the table hard to read; please fix the layout.
  4. [§3.3] In Eqs. (2)–(4), the notation y^{p}_{<k} is used before it is explicitly defined; defining it at first use would improve readability.

Circularity Check

1 steps flagged · score 4.0 of 10

Evaluation of CSL-Daily/Phoenix-2014T uses the authors' own pose-fitting outputs as both training targets and DTW ground truth, so the reported SOTA on those two benchmarks partly measures reproduction of the fitting pipeline.

  1. other [Section 3.1 and Section 4 (Datasets and Evaluation Metrics); Table 1]
    "To train SOKE, we use the SMPL-X poses of How2Sign dataset as provided in [3] along with the curated poses of CSL-Daily and Phoenix-2014T. ... we employ the well-established dynamic time warping (DTW) [2, 3] on both procrustes-aligned and original joint position errors (JPE) to measure sequence-level distances between the generated signs and ground truth."

    The 'ground truth' for CSL-Daily and Phoenix-2014T is the set of SMPL-X poses that the authors themselves fitted in Section 3.1 using OSX, WiLoR, and Mediapipe. Those same fitted poses are the regression targets of the decoupled tokenizer and the autoregressive generator: the training loss is computed on tokenized versions of these poses, and the DTW evaluation compares generated reconstructions against the same fitted poses. The reported DTW-JPE values (1.71 for CSL-Daily, 1.38 for Phoenix-2014T) therefore measure how well the model reproduces the author's pose-fitting pipeline, including any systematic fitting artifacts, rather than an independent sign-language ground truth.

full rationale

The paper's central derivation is self-contained: the decoupled VQ-VAE tokenizer, the autoregressive multilingual generator with multi-head decoding, and the retrieval-augmented condition are all specified with explicit loss functions and equations (Eqs. 1-4) that do not reduce to the predicted outputs. No equation-level self-reference was found. The circularity lies in evaluation: for two of the three benchmarks (CSL-Daily and Phoenix-2014T), the SMPL-X ground truth is curated by the authors' own pose-fitting pipeline in Section 3.1, and the same fitted poses serve as training regression targets (Section 4, Datasets). The DTW metric thus partially measures how faithfully the model reproduces the fitting pipeline's outputs, including its systematic errors (e.g., Mediapipe upper-body refinement and WiLoR hand substitution). The SGNify validation in Table S5 is external on a different dataset and protocol, so it does not establish that the CSL/Phoenix fitted targets are free of dataset-specific bias. However, the How2Sign benchmark uses independently provided SMPL-X poses from [3], and the user study with professional signers as well as back-translation BLEU scores provide external evidence that the generations carry linguistic content. The paper's Limitations section explicitly acknowledges that 'some reconstruction errors are inevitable,' and the evaluation methodology is transparent about the source of the labels. Since the core method does not depend on self-citation chains or renaming known results, and the evaluation circularity does not invalidate the method's internal consistency, a score of 4 reflects partial circularity in the external validity of the SOTA claim on CSL-Daily and Phoenix-2014T, not in the derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method is empirical, so the ledger mostly captures assumptions about data quality and representation adequacy. No new physical or theoretical entities are introduced; the motion token vocabulary is a learned compression of existing SMPL-X parameters.

free parameters (3)
  • lambda (embedding fusion weight) = 1/3
    Chosen in Table S7 as best on all three test sets; controls the weighted average of body and hand embeddings in multi-head decoding.
  • body codebook size N_B = 96
    Chosen via Table S6 based on PA-MPJPE on test splits; affects tokenizer reconstruction and downstream generation.
  • hand codebook sizes N_LH, N_RH = 192 each
    Chosen via Table S6 based on reconstruction and generation metrics; hand codebook size influences sign fidelity.
assumptions (5)
  • domain assumption The SMPL-X representation with 133 parameters, including upper body, hands and 10 expression parameters, captures enough sign-language semantics for generation.
    Section 3.1 defines the pose representation; facial expressions and non-manual markers beyond 10 expression parameters are not explicitly modeled.
  • ad hoc to paper The fitted SMPL-X poses for CSL-Daily and Phoenix-2014T are accurate enough to serve as training and evaluation labels.
    Section 3.1 and Supplementary Section A; only qualitative checks are shown on those datasets, with quantitative validation on the separate SGNify benchmark.
  • domain assumption Back-translation BLEU-4 from an unspecified sign-to-text model is a valid semantic fidelity measure for generated signs.
    Section 4, Evaluation Metrics; no details about the back-translator or its training data are given.
  • domain assumption Word-level dictionary signs tokenized by the same DETO provide useful conditioning despite sentence-level co-articulation.
    Section 3.4; the effect is measured by ablation but the mechanism is assumed rather than derived.
  • domain assumption The pretrained mBART-large-cc25 multilingual model transfers to sign token sequences when fine-tuned.
    Section 3.3 and Implementation Details; no analysis of the alignment between text and motion token spaces is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator." pith.science (2026). https://pith.science/paper/NRD5DXIB

@misc{pith2026241117799,
  author       = {Pith},
  title        = {Pith review of: Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRD5DXIB}},
  note         = {Machine review of arXiv:2411.17799}
}
read the original abstract

Sign language is a visual language that encompasses all linguistic features of natural languages and serves as the primary communication method for the deaf and hard-of-hearing communities. Although many studies have successfully adapted pretrained language models (LMs) for sign language translation (sign-to-text), the reverse task-sign language generation (text-to-sign)-remains largely unexplored. In this work, we introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs using a pretrained LM. To align sign language with the LM, we leverage a decoupled tokenizer that discretizes continuous signs into token sequences representing various body parts. During decoding, unlike existing approaches that flatten all part-wise tokens into a single sequence and predict one token at a time, we propose a multi-head decoding method capable of predicting multiple tokens simultaneously. This approach improves inference efficiency while maintaining effective information fusion across different body parts. To further ease the generation process, we propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs as auxiliary conditions, significantly improving the precision of generated signs. Extensive qualitative and quantitative evaluations demonstrate the effectiveness of SOKE.

Figures

Figures reproduced from arXiv: 2411.17799 by the authors.

Figure 1
Figure 1. We propose Signs as Tokens (SOKE), a unified sign language generator that can produce multilingual signs from text inputs. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of our proposed method, Signs as Tokens (SOKE). We begin by training a VQ-VAE-based decoupled tokenizer to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of various decoding methods. (a) Sequential decoding flattens all motion tokens into a single sequence, requiring [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of retrieval-enhanced SLG. We utilize exter [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons of generated signs between our proposed method, SOKE, with the SOTA method, S-MotionGPT [ [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative ablation study for retrieval-enhanced SLG. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: User study with professional ASL and CSL signers. We [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 68 canonical work pages

  1. [1]

    BSL-1K: Scaling up co-articulated sign language recogni- tion using mouthing cues

    Samuel Albanie, G ¨ul Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. BSL-1K: Scaling up co-articulated sign language recogni- tion using mouthing cues. In ECCV, pages 35–53, 2020. 15

    2020

  2. [2]

    Ham2pose: Animating sign language notation into pose se- quences

    Rotem Shalev Arkushin, Amit Moryossef, and Ohad Fried. Ham2pose: Animating sign language notation into pose se- quences. In CVPR, pages 21046–21056, 2023. 2, 5

    2023

  3. [3]

    Neural sign actors: A diffusion model for 3d sign language production from text

    Vasileios Baltatzis, Rolandos Alexandros Potamias, Evange- los Ververas, Guanxiong Sun, Jiankang Deng, and Stefanos Zafeiriou. Neural sign actors: A diffusion model for 3d sign language production from text. In CVPR, pages 1985–1995,

    1985

  4. [4]

    Neural sign language trans- lation

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Her- mann Ney, and Richard Bowden. Neural sign language trans- lation. In CVPR, 2018. 2, 3, 5, 13

    2018

  5. [5]

    Bench- marking large language models in retrieval-augmented gen- eration

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Bench- marking large language models in retrieval-augmented gen- eration. In AAAI, pages 17754–17762, 2024. 2, 5

    2024

  6. [6]

    Executing your commands via mo- tion diffusion in latent space

    Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via mo- tion diffusion in latent space. In CVPR, pages 18000–18010,

  7. [7]

    A simple multi-modality transfer learning base- line for sign language translation

    Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning base- line for sign language translation. In CVPR, pages 5120– 5130, 2022. 5

    2022

  8. [8]

    Two-stream network for sign language recognition and translation

    Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. In NeurIPS, 2022. 5

    2022

Show all 96 references
  1. [9]

    Realgen: Retrieval augmented generation for controllable traffic scenarios

    Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone. Realgen: Retrieval augmented generation for controllable traffic scenarios. In ECCV, pages 93–110, 2024. 2

    2024

  2. [10]

    Signavatar: Sign language 3d mo- tion reconstruction and generation

    Lu Dong, Lipisha Chaudhary, Fei Xu, Xiao Wang, Mason Lary, and Ifeoma Nwogu. Signavatar: Sign language 3d mo- tion reconstruction and generation. In FG, 2024. 2

    2024

  3. [11]

    How2sign: a large-scale multi- modal dataset for continuous american sign language

    Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2sign: a large-scale multi- modal dataset for continuous american sign language. In CVPR, pages 2735–2744, 2021. 2, 3, 5, 15

    2021

  4. [12]

    A sur- vey on rag meeting llms: Towards retrieval-augmented large language models

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A sur- vey on rag meeting llms: Towards retrieval-augmented large language models. In SIGKDD, pages 6491–6501, 2024. 2, 5

    2024

  5. [13]

    SignDiff: Learning diffusion models for American sign lan- guage production

    Sen Fang, Chunyu Sui, Xuedong Zhang, and Yapeng Tian. SignDiff: Learning diffusion models for American sign lan- guage production. arXiv preprint arXiv:2308.16082, 2023. 1, 2

    2023 arXiv

  6. [14]

    Signllm: Sign languages production large language models

    Sen Fang, Lei Wang, Ce Zheng, Yapeng Tian, and Chen Chen. Signllm: Sign languages production large language models. arXiv preprint arXiv:2405.10718, 2024. 2

    2024 arXiv

  7. [15]

    Collaborative regression of expressive bodies using moderation

    Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collaborative regression of expressive bodies using moderation. In3DV, pages 792–804,

  8. [16]

    Reconstructing signing avatars from video using linguistic priors

    Maria-Paola Forte, Peter Kulits, Chun-Hao P Huang, Vasileios Choutas, Dimitrios Tzionas, Katherine J Kuchen- becker, and Michael J Black. Reconstructing signing avatars from video using linguistic priors. In CVPR, pages 12791– 12801, 2023. 13

    2023

  9. [17]

    https://github.com/fxsjy/jieba

    fxsjy. https://github.com/fxsjy/jieba. In Github, 2012. 5

    2012

  10. [18]

    Cpr: Retrieval augmented generation for copyright protec- tion

    Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu- Xiang Wang, Ashwin Swaminathan, and Stefano Soatto. Cpr: Retrieval augmented generation for copyright protec- tion. In CVPR, pages 12374–12384, 2024. 2

    2024

  11. [19]

    Llms are good sign language translators

    Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. Llms are good sign language translators. In CVPR, pages 18362–18372, 2024. 1

    2024

  12. [20]

    Tm2t: Stochastic and tokenized modeling for the reciprocal gener- ation of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal gener- ation of 3d human motions and texts. In ECCV, pages 580– 597, 2022. 2

    2022

  13. [21]

    Scaling laws for autoregressive generative modeling

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. 1

    2010 arXiv

  14. [22]

    Hand-model- aware sign language recognition

    Hezhen Hu, Wengang Zhou, and Houqiang Li. Hand-model- aware sign language recognition. InAAAI, pages 1558–1566,

  15. [23]

    SignBERT+: Hand-model-aware self-supervised pre- training for sign language understanding

    Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. SignBERT+: Hand-model-aware self-supervised pre- training for sign language understanding. TPAMI, 2023. 1

    2023

  16. [24]

    To- wards fast and high-quality sign language production

    Wencan Huang, Wenwen Pan, Zhou Zhao, and Qi Tian. To- wards fast and high-quality sign language production. In MM, pages 3172–3181, 2021. 2

    2021

  17. [25]

    Dualsign: Semi-supervised sign language produc- tion with balanced multi-modal multi-task dual transforma- tion

    Wencan Huang, Zhou Zhao, Jinzheng He, and Mingmin Zhang. Dualsign: Semi-supervised sign language produc- tion with balanced multi-modal multi-task dual transforma- tion. In MM, pages 5486–5495, 2022. 2

    2022

  18. [26]

    Non- autoregressive sign language production with gaussian space

    Euijun Hwang, Jung-Ho Kim, and Jong C Park. Non- autoregressive sign language production with gaussian space. In BMVC, 2021. 7

    2021

  19. [27]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. NeurIPS, 36:20067–20079, 2023. 2, 4, 6, 7, 13, 14

    2023

  20. [28]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. Active retrieval augmented generation. In EMNLP, pages 7969–7992, 2023. 2 9

    2023

  21. [29]

    Cosign: Exploring co-occurrence signals in skeleton-based continuous sign language recognition

    Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang, Lei Lei, and Xilin Chen. Cosign: Exploring co-occurrence signals in skeleton-based continuous sign language recognition. In ICCV, pages 20676–20686, 2023. 2, 3

    2023

  22. [30]

    Visual alignment pre-training for sign language translation

    Peiqi Jiao, Yuecong Min, and Xilin Chen. Visual alignment pre-training for sign language translation. In ECCV, pages 349–367, 2024. 5

    2024

  23. [31]

    isign: A benchmark for indian sign lan- guage processing

    Abhinav Joshi, Romit Mohanty, Mounika Kanakanti, An- desha Mangla, Sudeep Choudhary, Monali Barbate, and Ashutosh Modi. isign: A benchmark for indian sign lan- guage processing. In Findings of ACL, pages 10827–10844,

  24. [32]

    MS-ASL: A large- scale data set and benchmark for understanding American sign language

    Hamid Reza Vaezi Joze and Oscar Koller. MS-ASL: A large- scale data set and benchmark for understanding American sign language. In BMVC, 2019. 5

    2019

  25. [33]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  26. [34]

    Continuous sign language recognition: Towards large vocabulary statis- tical recognition systems handling multiple signers

    Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocabulary statis- tical recognition systems handling multiple signers. CVIU, 141:108–125, 2015. 15

    2015

  27. [35]

    Introducing the DW-DGS – the digital dictionary of DGS

    Gabriele Langer, Anke M ¨uller, Sabrina W¨ahl, Felicitas Otte, Lea Sepke, and Thomas Hanke. Introducing the DW-DGS – the digital dictionary of DGS. In Proceedings of the LREC- COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign ...

    2024

  28. [36]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS, 33:9459–9474, 2020. 2

    2020

  29. [37]

    Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison

    Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In WACV, pages 1459–1469, 2020. 5

    2020

  30. [38]

    Uni-sign: Toward unified sign language understanding at scale

    Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, and Houqiang Li. Uni-sign: Toward unified sign language understanding at scale. In ICLR, 2025. 1

    2025

  31. [39]

    One-stage 3d whole-body mesh recovery with component aware transformer

    Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. In CVPR, pages 21159–21168, 2023. 3, 8, 13

    2023

  32. [40]

    Multilingual denoising pre-training for neural machine translation

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. TACL, 8:726–742, 2020. 2, 4, 5

    2020

  33. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 5

    2019

  34. [42]

    Humantomato: Text-aligned whole-body motion generation

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Humantomato: Text-aligned whole-body motion generation. InICML, pages 32939–32977, 2024. 2, 3, 4, 7

    2024

  35. [43]

    Medi- apipe: A framework for building perception pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Medi- apipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019. 3, 13

    1906 arXiv

  36. [44]

    Considerations for meaningful sign language machine translation based on glosses

    Mathias M ¨uller, Zifan Jiang, Amit Moryossef, Annette Rios Gonzales, and Sarah Ebling. Considerations for meaningful sign language machine translation based on glosses. In ACL, pages 682–693, 2023. 2

    2023

  37. [45]

    A Hong Kong sign language corpus collected from sign- interpreted tv news

    Zhe Niu, Ronglai Zuo, Brian Mak, and Fangyun Wei. A Hong Kong sign language corpus collected from sign- interpreted tv news. In LREC-COLING, pages 636–646,

  38. [46]

    BLEU: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002. 5

    2002

  39. [47]

    Expressive body capture: 3D hands, face, and body from a single image

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, pages 10975– 10985, 2019. 2, 13

    2019

  40. [48]

    Wilor: End-to-end 3d hand localization and reconstruction in-the-wild

    Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In CVPR, 2025. 3, 13

    2025

  41. [49]

    Signgen: End-to-end sign language video generation with latent diffusion

    Fan Qi, Yu Duan, Changsheng Xu, and Huaiwen Zhang. Signgen: End-to-end sign language video generation with latent diffusion. In ECCV, 2024. 1

    2024

  42. [50]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67, 2020. 6

    2020

  43. [51]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 2

    2022

  44. [52]

    Frankmo- cap: A monocular 3d whole-body pose estimation system via regression and integration

    Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmo- cap: A monocular 3d whole-body pose estimation system via regression and integration. InICCVW, pages 1749–1759,

  45. [53]

    Progressive transformers for end-to-end sign language pro- duction

    Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Progressive transformers for end-to-end sign language pro- duction. In ECCV, pages 687–705, 2020. 1, 2, 7

    2020

  46. [54]

    Adversarial training for multi-channel sign language produc- tion

    Ben Saunders, Necati Cihan Camg ¨oz, and Richard Bowden. Adversarial training for multi-channel sign language produc- tion. In BMVC, 2020. 7

    2020

  47. [55]

    Continuous 3D multi-channel sign language production via progressive transformers and mixture density networks

    Ben Saunders, Necati Cihan Camgoz, and Richard Bow- den. Continuous 3D multi-channel sign language production via progressive transformers and mixture density networks. IJCV, 129(7):2113–2135, 2021. 2

    2021

  48. [56]

    Signing at scale: Learning to co-articulate signs for large- scale photo-realistic sign language production

    Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Signing at scale: Learning to co-articulate signs for large- scale photo-realistic sign language production. In CVPR, pages 5141–5151, 2022. 1, 2, 3

    2022

  49. [57]

    Human motion diffusion as a generative prior

    Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as a generative prior. In ICLR, 2024. 2 10

    2024

  50. [58]

    knn- diffusion: Image generation via large-scale retrieval

    Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. knn- diffusion: Image generation via large-scale retrieval. In ICLR, 2023. 3

    2023

  51. [59]

    Pose-guided fine-grained sign lan- guage video generation

    Tongkai Shi, Lianyu Hu, Fanhua Shang, Jichao Feng, Pei- dong Liu, and Wei Feng. Pose-guided fine-grained sign lan- guage video generation. In ECCV, pages 392–409, 2024. 1

    2024

  52. [60]

    There and back again: 3d sign language generation from text using back-translation

    Stephanie Stoll, Armin Mustafa, and Jean-Yves Guillemaut. There and back again: 3d sign language generation from text using back-translation. In 3DV, pages 187–196, 2022. 7

    2022

  53. [61]

    Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In ICLR, 2024. 2

    2024

  54. [62]

    Multilin- gual gloss-free sign language translation: Towards building a sign language foundation model

    Sihan Tan, Taro Miyazaki, and Kazuhiro Nakadai. Multilin- gual gloss-free sign language translation: Towards building a sign language foundation model. In ACL, 2025. 1

    2025

  55. [63]

    Gloss semantic-enhanced network with online back- translation for sign language production

    Shengeng Tang, Richang Hong, Dan Guo, and Meng Wang. Gloss semantic-enhanced network with online back- translation for sign language production. In MM, pages 5630–5638, 2022. 2

    2022

  56. [64]

    Discrete to continuous: Generating smooth transition poses from sign language observations

    Shengeng Tang, Jiayi He, Lechao Cheng, Jingjing Wu, Dan Guo, and Richang Hong. Discrete to continuous: Generating smooth transition poses from sign language observations. In CVPR, 2025. 1

    2025

  57. [65]

    Sign-idd: Iconicity disentangled diffu- sion for sign language production

    Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, and Richang Hong. Sign-idd: Iconicity disentangled diffu- sion for sign language production. In AAAI, 2025. 1

    2025

  58. [66]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024. 1

    2024

  59. [67]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017. 3

    2017

  60. [68]

    Searching for best practices in retrieval-augmented generation

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for best practices in retrieval-augmented generation. In EMNLP, pages 17716– 17736, 2024. 2, 5

    2024

  61. [69]

    Improving continuous sign language recognition with cross-lingual signs

    Fangyun Wei and Yutong Chen. Improving continuous sign language recognition with cross-lingual signs. In ICCV, pages 23612–23621, 2023. 5, 8

    2023

  62. [70]

    Sign2gpt: Leveraging large language models for gloss-free sign language translation

    Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2gpt: Leveraging large language models for gloss-free sign language translation. In ICLR, 2024. 1

    2024

  63. [71]

    G2p-ddm: Generating sign pose sequence from gloss sequence with discrete diffusion model

    Pan Xie, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, and Zexian Li. G2p-ddm: Generating sign pose sequence from gloss sequence with discrete diffusion model. In AAAI, pages 6234–6242, 2024. 2, 3

    2024

  64. [72]

    Retrieval-augmented egocentric video captioning

    Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval-augmented egocentric video captioning. In CVPR, pages 13525–13536, 2024. 2

    2024

  65. [73]

    Sign language translation with it- erative prototype

    Huijie Yao, Wengang Zhou, Hao Feng, Hezhen Hu, Hao Zhou, and Houqiang Li. Sign language translation with it- erative prototype. In ICCV, pages 15592–15601, 2023. 1

    2023

  66. [74]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. InCVPR, pages 469–480, 2023. 2, 3, 4, 7

    2023

  67. [75]

    MLSLT: Towards multilingual sign language translation

    Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. MLSLT: Towards multilingual sign language translation. In CVPR, pages 5109–5119, 2022. 1

    2022

  68. [76]

    T2S-GPT: Dynamic vector quantization for au- toregressive sign language production from text

    Aoxiong Yin, Haoyuan Li, Kai Shen, Siliang Tang, and Yuet- ing Zhuang. T2S-GPT: Dynamic vector quantization for au- toregressive sign language production from text. In ACL,

  69. [77]

    Including signed languages in natural language processing

    Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Gold- berg, and Malihe Alikhani. Including signed languages in natural language processing. In ACL, pages 7347–7360,

  70. [78]

    Signavatars: A large-scale 3d sign language holis- tic motion dataset and benchmark

    Zhengdi Yu, Shaoli Huang, Yongkang Cheng, and Tolga Birdal. Signavatars: A large-scale 3d sign language holis- tic motion dataset and benchmark. In ECCV, pages 1–19,

  71. [79]

    Neural sign language synthe- sis: Words are our glosses

    Jan Zelinka and Jakub Kanis. Neural sign language synthe- sis: Words are our glosses. In WACV, pages 3395–3403,

  72. [80]

    Pymaf-x: To- wards well-aligned full-body model regression from monoc- ular images

    Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. Pymaf-x: To- wards well-aligned full-body model regression from monoc- ular images. IEEE TPAMI, 45(10):12287–12303, 2023. 13

    2023

  73. [81]

    Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In CVPR, pages 14730–14740,

  74. [82]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. IEEE TPAMI, 2024. 2

    2024

  75. [83]

    Mo- tiongpt: Finetuned llms are general-purpose motion genera- tors

    Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Mo- tiongpt: Finetuned llms are general-purpose motion genera- tors. In AAAI, pages 7368–7376, 2024. 2

    2024

  76. [84]

    BEST: BERT pre-training for sign language recognition with coupling tokenization

    Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, and Houqiang Li. BEST: BERT pre-training for sign language recognition with coupling tokenization. In AAAI, 2023. 3

    2023

  77. [85]

    Masa: Motion-aware masked autoencoder with semantic alignment for sign language recognition

    Weichao Zhao, Hezhen Hu, Wengang Zhou, Yunyao Mao, Min Wang, and Houqiang Li. Masa: Motion-aware masked autoencoder with semantic alignment for sign language recognition. IEEE TCSVT, 2024. 5

    2024

  78. [86]

    Self-supervised representation learning with spatial-temporal consistency for sign language recognition

    Weichao Zhao, Wengang Zhou, Hezhen Hu, Min Wang, and Houqiang Li. Self-supervised representation learning with spatial-temporal consistency for sign language recognition. IEEE TIP, 2024. 3

    2024

  79. [87]

    Gloss-free sign language translation: Improving from visual- language pretraining

    Benjia Zhou, Zhigang Chen, Albert Clap ´es, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual- language pretraining. In ICCV, pages 20871–20881, 2023. 1

    2023

  80. [88]

    Spatial-temporal multi-cue network for continuous sign lan- guage recognition

    Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for continuous sign lan- guage recognition. In AAAI, pages 13009–13016, 2020. 2, 3 11

    2020

  81. [89]

    Improving sign language translation with monolingual data by sign back-translation

    Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In CVPR, 2021. 2, 3, 5, 13, 15

    2021

  82. [90]

    Spatial-temporal multi-cue network for sign language recog- nition and translation

    Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recog- nition and translation. TMM, pages 1–1, 2021. 1

    2021

  83. [91]

    Avatargpt: All- in-one framework for motion understanding planning gener- ation and beyond

    Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All- in-one framework for motion understanding planning gener- ation and beyond. In CVPR, pages 1357–1366, 2024. 2

    2024

  84. [92]

    C2SLR: Consistency-enhanced continuous sign language recognition

    Ronglai Zuo and Brian Mak. C2SLR: Consistency-enhanced continuous sign language recognition. In CVPR, pages 5131–5140, 2022. 2

    2022

  85. [93]

    Improving continuous sign lan- guage recognition with consistency constraints and signer re- moval

    Ronglai Zuo and Brian Mak. Improving continuous sign lan- guage recognition with consistency constraints and signer re- moval. ACM TOMM, 2024. 3

    2024

  86. [94]

    Natural language-assisted sign language recognition

    Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural language-assisted sign language recognition. In CVPR, pages 14890–14900, 2023. 5

    2023

  87. [95]

    A simple baseline for spoken lan- guage to sign language translation with 3d avatars

    Ronglai Zuo, Fangyun Wei, Zenggui Chen, Brian Mak, Jiao- long Yang, and Xin Tong. A simple baseline for spoken lan- guage to sign language translation with 3d avatars. InECCV,

  88. [96]

    Towards on- line continuous sign language recognition and translation

    Ronglai Zuo, Fangyun Wei, and Brian Mak. Towards on- line continuous sign language recognition and translation. In EMNLP, 2024. 5 12 Ground TruthRecon. Mesh Figure S8. Qualitative comparisons between ground truth video frames and reconstructed meshes obtained from the proposed...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.