{"id":"07b70ab0-9763-4f36-b58c-34625fe54ce5","arxiv_id":"2411.17799","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SOKE generates 3D sign-language avatars from text across ASL, CSL and DGS using a pretrained multilingual language model, multi-head decoding, and retrieval of dictionary sign tokens.","lead":"This paper introduces SOKE, a system that turns written text into 3D sign-language avatars for American, Chinese, and German sign language, using a pretrained language model that predicts motion tokens for the body and both hands. It adds a retrieval step that pulls word-level signs from dictionaries to improve accuracy, and reports lower pose errors than prior methods on three benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation of the CSL-Daily/Phoenix-2014T generations uses fitted poses as ground truth for the same pose-estimation pipeline that produced the training targets; a correlated-error artifact could inflate the reported DTW gains, and the SGNify validation in Table S5 does not rule this out.","rationale":"The reader's weakest assumption is that the author-fitted SMPL-X poses for CSL-Daily and Phoenix-2014T are accurate enough to serve as both training targets and DTW ground truth; I agree this is the most load-bearing issue. Section 3.1 describes the fitting pipeline: OSX body estimation plus WiLoR hands plus Mediapipe-2D-driven upper-body refinement, with a weak-perspective projection loss and temporal/pose regularization. The evaluation metric (DTW-PA-JPE) is computed on the same pose representation that the generator was trained to reproduce. Any systematic bias in the fitting pipeline is therefore shared between training target and test ground truth, and the likelihood of such bias is high: sign language hands and arms are notoriously difficult to reconstruct, the fitting only uses 2D keypoints for the upper-body refinement, and no quantitative per-dataset validation is provided for CSL-Daily or Phoenix-2014T. Table S5 validates on SGNify, which is a different dataset with its own motion capture poses, and even there the body error (46.73 mm) is only comparable to, not better than, NSA. For the central SOTA claim to hold, the fitted poses must be good enough that DTW gains reflect actual sign quality rather than artifact consistency; that condition is not established. I also note that the BLEU-4 back-translation scores and user study qualitatively point in the same direction and provide some independent support, but back-translation models in this setting are trained on the same fitted poses, so they may share the same artifact correlation; the user study is on 15 samples and is not sufficient to rule out the concern. Thus the appropriate verdict remains conditional: the method is plausible and internally coherent, but the SOTA claim on two of three benchmarks should be treated as provisional until independent pose validation is performed. I agree with the reader's identification of the concern, so agreement_with_reader is 'agree' and I recommend keeping the CONDITIONAL verdict.","tokens_in":18763,"tokens_out":1883,"duration_ms":14885,"concrete_test":"Independently fit a subset (e.g., 200 held-out clips) of CSL-Daily and Phoenix-2014T with an alternative whole-body fitting pipeline (for example, a different hand reconstruction method), and re-run the SOKE evaluation on those clips using the alternative poses as ground truth. If the DTW-PA-JPE body/hand errors change by more than ~15% relative, or if the ranking against S-MotionGPT reverses on either dataset, the reported gains are substantially artifacts of the fitted targets. As a cheaper secondary check, compute the DTW error on the training set: a near-zero train DTW with large test DTW would suggest overfitting to fitted-target artifacts.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central SOTA claim rests on comparing generated SMPL-X motions against the author-fitted CSL-Daily and Phoenix-2014T poses, but those same fitted poses are the training regression targets. If pose-fitting artifacts (e.g., Mediapipe 2D-joint re-projection biases in the upper-body refinement in Section 3.1, or WiLoR hand-parameter substitution errors) are systematic, the model can learn to reproduce them during training, and the DTW metric will reward that reproduction even when the linguistic sign is wrong. The quantitative validation in Table S5 is on SGNify, a different benchmark with different signers, capture setup, and evaluation protocol (mean per-vertex mesh error), so it cannot establish that the CSL-Daily/Phoenix-2014T fitted targets are free of dataset-specific systematic errors. The paper's own Limitations section concedes reconstruction errors are inevitable, and Figure S8 is only qualitative. Therefore the reported DTW gains of 1.71 and 1.38 on CSL-Daily and Phoenix-2014T could partly measure consistency with the fitted targets rather than linguistic correctness. This does not challenge the internal consistency of the method, but it does weaken the external validity of the SOTA claim on two of the three benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SOKE, a unified multilingual sign language generator from text to 3D avatar motion. The method has two stages: a decoupled VQ-VAE tokenizer that discretizes upper-body, left-hand, and right-hand motion into separate token streams, and an autoregressive multilingual generator built on mBART-large, which predicts the three token streams jointly via a multi-head decoding strategy. The generator is conditioned on text and on motion tokens retrieved from external word-level sign dictionaries. The authors evaluate on How2Sign, CSL-Daily, and Phoenix-2014T, reporting state-of-the-art DTW-PA-JPE values of 2.35, 1.71, and 1.38 and back-translation BLEU-4 scores of 14.48, 11.30, and 11.87. Ablations cover decoding strategies, retrieval, multilingual scaling, tokenizer decomposition, and codebook sizes, and a small user study with professional signers is included. For CSL-Daily and Phoenix-2014T, the SMPL-X ground-truth poses are produced by the authors' own OSX/WiLoR/Mediapipe fitting pipeline.","tokens_in":19206,"tokens_out":3369,"duration_ms":32568,"significance":"The paper's core idea—treating sign language production as tokenized autoregressive prediction with a pretrained multilingual LM—is timely and technically coherent. The decoupled tokenizer, the multi-head decoding scheme that reduces decoding steps from O(3K) to O(K), and the retrieval mechanism that uses dictionary motion tokens as conditions rather than splicing them into the output are all reasonable design contributions. The ablations are internally consistent: multi-head decoding improves over both sequential and parallel decoding, retrieval consistently reduces DTW error, and multilingual training helps. If the reported gains are externally valid, SOKE would be a meaningful step toward practical multilingual text-to-sign systems. The main risk is that two of the three benchmark evaluations use labels produced by the same pose-fitting pipeline that generated the training targets, so the DTW numbers on CSL-Daily and Phoenix-2014T may partly reflect consistency with fitted poses rather than linguistically correct signing. The quantitative validation in Table S5 is on a different dataset with a different protocol and therefore does not fully remove this concern.","major_comments":[{"comment":"The ground-truth SMPL-X poses for CSL-Daily and Phoenix-2014T are generated by the authors' OSX/WiLoR/Mediapipe fitting pipeline, and the same fitted poses serve simultaneously as training regression targets and as DTW evaluation labels. If the fitting pipeline has systematic dataset-specific errors (e.g., Mediapipe-based upper-body refinement biases or WiLoR hand-parameter substitution artifacts), the model can learn to reproduce those errors during training, and the DTW metric will reward that reproduction even when the produced sign is linguistically wrong. Table S5 validates the pipeline on SGNify, which has different signers, capture setup, and an evaluation metric (mean per-vertex error) different from the DTW protocol used in Table 1; it therefore cannot establish that the CSL-Daily and Phoenix-2014T fitted targets are free of dataset-specific systematic errors. The paper's own Limitations section concedes that reconstruction errors are inevitable, and Figure S8 is only qualitative. This is load-bearing because the central SOTA claim on two of the three benchmarks depends on the accuracy of these fitted labels. Please provide a quantitative validation of the fitted poses on CSL-Daily and Phoenix-2014T (e.g., against manual joint annotations, alternative estimation methods, or signer evaluation), or evaluate the generated signs with at least one protocol that does not use the same fitted poses as ground truth (e.g., an independent sign-language recognizer or human ratings for DGS/CSL).","section":"§3.1, Tables 1 and S5, Figure S8"},{"comment":"None of the quantitative claims in Tables 1 and 2 is accompanied by error bars, bootstrap intervals, or significance tests. The text repeatedly uses 'significantly lower' and 'significantly enhances', but all numbers appear to come from a single run or a single evaluation pass. Given that the comparisons include reimplemented baselines and that the central claim is state-of-the-art performance, the absence of variance estimates makes it impossible to assess whether the reported margins are robust. Please report results over multiple random seeds or bootstrap resampling, and where relevant perform a paired significance test.","section":"§4.1, Tables 1 and 2"},{"comment":"The retrieval dictionary is filtered by retaining, for each word, only the instance with the lowest reconstruction error after passing through the tokenizer. Because the tokenizer is trained on the same distribution and the same fitting pipeline as the evaluation data, this selection criterion can systematically favor entries that are easiest for the model to reconstruct, which may inflate the retrieval gains reported in Table 2. The paper does not report dictionary coverage, retrieval frequency, or an ablation in which unfiltered or randomly selected dictionary instances are used. Please provide these analyses to show that the retrieval benefit is not an artifact of the selection rule.","section":"§3.4"}],"minor_comments":[{"comment":"The user study involves only 9 professional signers and 15 generated signs per method, and no statistical test is reported for the rating differences. A paired test or confidence intervals would strengthen the claim that the ratings differ beyond noise.","section":"§4.3, Figure 7"},{"comment":"The typo 'addtional' appears in the Related Work section; please correct it.","section":"§2"},{"comment":"The header of Table S6 contains a formatting error ('N B Z N LH Z = N RH Z' is repeated and interspersed with column values), making the table hard to read; please fix the layout.","section":"Table S6"},{"comment":"In Eqs. (2)–(4), the notation y^{p}_{<k} is used before it is explicitly defined; defining it at first use would improve readability.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a strong engineering contribution, but its headline SOTA claims on CSL-Daily and Phoenix-2014T are weaker than they look. The method itself is sound: a decoupled VQ-VAE tokenizer, multi-head decoding that predicts body and hand tokens simultaneously, and retrieval conditioning from sign dictionaries. The multi-head decoding is the genuine new piece, and Table 2 shows it beats sequential decoding in quality while cutting latency by half, and beats parallel decoding on both axes. The retrieval ablation is clean, and the multilingual scaling experiment is a nice demonstration. The user study with professional signers, though small, gives the results a plausibility anchor that pure metrics lack.\n\nThe soft spot is exactly what the stress-test note says. For CSL-Daily and Phoenix-2014T, the same OSX + WiLoR + Mediapipe fitting pipeline generates both the training targets and the DTW ground truth. If the fitting has systematic errors, the model can learn to reproduce them and the DTW metric will reward that, inflating the reported 1.71 and 1.38 errors. The SGNify validation in Table S5 does not rule this out because it is a different dataset with a different capture setup, and the paper's own Limitations section admits reconstruction errors are inevitable. The How2Sign results are on poses from the separate NSA pipeline, so they are less exposed, but all three benchmarks lack error bars or significance tests.\n\nThat said, this should not sink the paper. The BLEU back-translation scores and the user study are external corroboration, and the core method is transparent and internally consistent. The concern is about the magnitude of the SOTA claim, not about whether the approach works. A serious referee should ask for independent validation on at least one fitted-pose benchmark, or for a release of the fitting pipeline and the curated poses so others can compare. The paper is well written, the ablations are honest, and the limitations are acknowledged in good faith.\n\nWho gets value: anyone working on sign language production or embodied motion generation. It deserves a serious referee and, with revision to address the evaluation circularity, is publishable. I would not cite the CSL/Phoenix numbers as definitive before that revision, but I would cite the multi-head decoding idea.","headline":"Solid, honest engineering paper with a real efficiency win, but the CSL/Phoenix SOTA numbers should be read with caution because the ground truth and training labels come from the same pose-fitting pipeline.","tokens_in":19580,"tokens_out":1999,"would_cite":false,"duration_ms":19922,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sign language generation becomes autoregressive token prediction.","keywords":["sign language generation","text-to-sign","multilingual language model","decoupled tokenizer","multi-head decoding","retrieval-enhanced generation","SMPL-X avatar","VQ-VAE"],"falsifier":"Fitting the same pipeline on a mocap or mesh-annotated dataset of Chinese and German continuous signs would settle it: if hand or body fitting errors on those languages are as large as the reported DTW differences between SOKE and its closest baseline, the DTW gains measure fit consistency rather than sign correctness. A blind rating by native signers of whether generated signs convey the source text, comparing SOKE with S-MotionGPT on the same examples, would test the linguistic claim directly.","tokens_in":18596,"feed_emoji":"🤟","tokens_out":8128,"duration_ms":63744,"temperature":0.7,"pith_summary":"The paper argues that sign language generation—turning a written sentence into a 3D signing avatar—should be framed as a language modeling problem rather than a visual generation problem. It introduces SOKE, a single multilingual generator for American, Chinese, and German sign languages, and claims that the discrete, multi-cue structure of signs can be captured by a decoupled tokenizer that turns continuous motion into separate token streams for the upper body and each hand. A pretrained multilingual language model then generates those tokens autoregressively from text, helped by retrieved word-level dictionary signs. If this claim is right, text-to-sign can be handled by one scalable model that improves with more data and more languages, rather than by a separate model per sign language or per dataset.","feed_headline":"One model signs in three languages at once","feed_subtitle":"SOKE turns text into 3D sign avatars and beats prior methods on three benchmarks in one model.","key_machinery":"The mechanism that carries the argument is the decoupled tokenizer combined with multi-head decoding and retrieval-conditioned prompting. The decoupled tokenizer discretizes continuous SMPL-X sign motions into three independent token streams (upper body codebook of 96 codes, left-hand and right-hand codebooks of 192 codes each), so the model can represent the simultaneous body and hand cues that carry sign semantics. The multi-head decoder predicts all three part-wise tokens at once at each step, with input embeddings fused as $E=(1-2\\lambda)E_B+\\lambda E_{LH}+\\lambda E_{RH}$; this cuts the decoding steps from $3K$ to $K$ while still sharing context across body parts. The retrieval-enhanced condition prepends the motion tokens of lemmatized dictionary words to the text prompt, giving the encoder precise word-level signs to anchor sentence-level generation.","core_discovery":"On its own terms, the paper's central discovery is that sign language production can be reduced to next-token prediction over three parallel part-wise token streams. SOKE trains three VQ-VAE codebooks—one for the upper body, one for the left hand, one for the right hand—to discretize SMPL-X sign motions, then inserts these tokens into the vocabulary of a multilingual language model. At each decoding step three heads predict the body and hand tokens simultaneously, with the decoder input at each step set to the weighted average embedding $E = (1-2\\lambda)E_B+\\lambda E_{LH}+\\lambda E_{RH}$ with $\\lambda = 1/3$. Word-level dictionary signs, converted to motion tokens, are appended to the text prompt as auxiliary conditions. The paper reports state-of-the-art procrustes-aligned DTW body errors of 2.35, 1.71, and 1.38 on How2Sign, CSL-Daily, and Phoenix-2014T, with back-translation BLEU-4 of 14.48, 11.30, and 11.87, and shows in ablations that multilingual training, the decoupled tokenizer, multi-head decoding, and retrieval each contribute.","pith_inferences":["A testable extension is to let native signers rate semantic equivalence of the generated signs directly against the source text; this would separate meaning fidelity from pose-fit consistency.","The paper's argument implies that the same tokenizer plus language-model recipe could work in reverse for sign-to-text translation, forming a truly bilingual text–sign model within one vocabulary.","Because the retrieval condition uses word-level dictionary signs, the expected gain should depend on dictionary coverage per language; comparing retrieval gains across languages with controlled dictionary sizes would test this dependence.","The reported DTW numbers on CSL-Daily and Phoenix-2014T measure agreement with the paper's fitted SMPL-X targets; if independently mocap-annotated data for those languages become available, re-running the comparison would tell whether the gains persist."],"forward_implications":["Text-to-sign generation does not need per-language models: a single fine-tuned multilingual language model can serve multiple sign languages.","Because encoding and decoding scale with data, adding more sign languages and larger sign dictionaries should keep improving generation quality and vocabulary coverage.","Multi-head decoding cuts inference steps from three times the motion length to the motion length itself, making real-time avatar signing a realistic deployment target.","Retrieval of word-level dictionary signs improves the precision of generated signs by around one-fifth on each benchmark without inserting unnatural dictionary poses directly into the output.","Separating upper-body, left-hand, and right-hand tokenization consistently outperforms whole-body tokenization, indicating that hand–body decomposition is central to sign semantics."],"supporting_citations":[{"why":"Supplies the How2Sign SMPL-X poses that SOKE trains on and the Neural Sign Actors baseline it outperforms, plus the upper-body pose-refinement protocol.","marker":"[3]"},{"why":"Provides the German Sign Language benchmark Phoenix-2014T on which SOKE reports its lowest DTW errors.","marker":"[4]"},{"why":"Provides the American Sign Language benchmark How2Sign and its pose annotations.","marker":"[11]"},{"why":"The MotionGPT approach SOKE adapts; reimplemented as S-MotionGPT, the closest baseline in comparisons and ablations.","marker":"[27]"},{"why":"OSX body mesh recovery, the first stage of the SMPL-X pose-fitting pipeline used to curate CSL-Daily and Phoenix-2014T poses.","marker":"[39]"},{"why":"The pretrained multilingual mBART model used as the generator backbone; its vocabulary and encoder-decoder structure carry the tokens.","marker":"[40]"},{"why":"Introduces the decoupled VQ-VAE tokenizer and the flattening sequential decoding method that multi-head decoding replaces.","marker":"[42]"},{"why":"WiLoR hand reconstruction, used to replace the hand parameters in the pose-fitting pipeline and crucial for hand-shape fidelity.","marker":"[48]"},{"why":"Baseline autoregressive sign production method (T2S-GPT) that the paper compares against and extends in spirit.","marker":"[76]"},{"why":"Provides the Chinese Sign Language benchmark CSL-Daily and the posed data used for training and evaluation.","marker":"[89]"}],"fun_headline_variants":["SOKE: multilingual model generates sign avatars from text","Three token streams, one LM: SOKE signs in 3 languages","Multi-head decoding: SOKE predicts body and hand tokens at once","Retrieval-augmented sign generation improves precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the SMPL-X poses fitted for CSL-Daily and Phoenix-2014T are accurate enough to serve as both training targets and DTW evaluation ground truth; the paper validates them qualitatively on those datasets and quantitatively only on a different benchmark, SGNify.","fun_headline_variants_meta":{"raw":{"variants":["SOKE: multilingual model generates sign avatars from text","Three token streams, one LM: SOKE signs in 3 languages","Multi-head decoding: SOKE predicts body and hand tokens at once","Retrieval-augmented sign generation improves precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001279,"raw_usage":{"total_tokens":5264,"prompt_tokens":1018,"completion_tokens":4246,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":4172}},"tokens_in":634,"tokens_out":4246,"duration_ms":28616,"temperature":1.0,"reasoning_tokens":4172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:52:00.201282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fitting the same pipeline on a mocap or mesh-annotated dataset of Chinese and German continuous signs would settle it: if hand or body fitting errors on those languages are as large as the reported DTW differences between SOKE and its closest baseline, the DTW gains measure fit consistency rather than sign correctness. A blind rating by native signers of whether generated signs convey the source text, comparing SOKE with S-MotionGPT on the same examples, would test the linguistic claim directly.","supporting_citations":[{"cited_title":"Motiongpt: Human motion as a foreign language","cited_arxiv_id":null,"evidence_quote":"The MotionGPT approach SOKE adapts; reimplemented as S-MotionGPT, the closest baseline in comparisons and ablations."},{"cited_title":"One-stage 3d whole-body mesh recovery with component aware transformer","cited_arxiv_id":null,"evidence_quote":"OSX body mesh recovery, the first stage of the SMPL-X pose-fitting pipeline used to curate CSL-Daily and Phoenix-2014T poses."},{"cited_title":"Multilingual denoising pre-training for neural machine translation","cited_arxiv_id":null,"evidence_quote":"The pretrained multilingual mBART model used as the generator backbone; its vocabulary and encoder-decoder structure carry the tokens."},{"cited_title":"Humantomato: Text-aligned whole-body motion generation","cited_arxiv_id":null,"evidence_quote":"Introduces the decoupled VQ-VAE tokenizer and the flattening sequential decoding method that multi-head decoding replaces."},{"cited_title":"Wilor: End-to-end 3d hand localization and reconstruction in-the-wild","cited_arxiv_id":null,"evidence_quote":"WiLoR hand reconstruction, used to replace the hand parameters in the pose-fitting pipeline and crucial for hand-shape fidelity."},{"cited_title":"T2S-GPT: Dynamic vector quantization for au- toregressive sign language production from text","cited_arxiv_id":null,"evidence_quote":"Baseline autoregressive sign production method (T2S-GPT) that the paper compares against and extends in spirit."},{"cited_title":"Improving sign language translation with monolingual data by sign back-translation","cited_arxiv_id":null,"evidence_quote":"Provides the Chinese Sign Language benchmark CSL-Daily and the posed data used for training and evaluation."}],"review_version":1}