{"id":"8d237ad1-6365-4389-95c8-3de9067cfc17","arxiv_id":"2608.09006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SignLlama uses filtered pseudo-gloss pretraining and visual-prioritized distillation to make Llama-based sign language translation competitive on four benchmarks.","lead":"This paper presents SignLlama, a way to make large language models translate sign language video into spoken sentences without gloss annotations. It combines a pretraining step that makes approximate label sequences from the target text with a distillation step that forces the model to rely on the video rather than on text hints.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing premise is that text-derived pseudo-glosses used in FPG-CTC match the temporal order of signed content; the paper provides no evidence, and gold glosses on Phoenix14T/CSL-Daily can settle it directly.","rationale":"The reader's weakest_assumption identifies exactly the premise I would stress-test. I did not find a more load-bearing flaw in the argument. The empirical structure is internally consistent: ablations (Tables 1, 2, 14) show both FPG-CTC and VPD contribute, and comparisons are against reasonable gloss-free baselines. The self-reported limitation in Section 10 and the poor pseudo-gloss WER in Appendix D make the temporal-fidelity question concrete rather than speculative. The most decisive available evidence is the gold gloss annotations on Phoenix14T and CSL-Daily, which the method never uses; they permit a direct, quantitative order-faithfulness test. The other concerns of the reader (missing code, no variance, unspecified masking implementation, and the SignDINO relationship) are real but secondary; the pseudo-gloss order issue is the one whose failure would void the mechanism behind the largest single gain in the paper. If the proposed alignment test comes back favorable, the correct disposition is to keep the conditional status and ask for code and reproducibility details; if it comes back unfavorable, the central mechanistic claim of FPG-CTC is not supported even though the method might still work for other reasons. I would keep the reader's CONDITIONAL verdict; no adjustment is needed.","tokens_in":19550,"tokens_out":7537,"duration_ms":75391,"concrete_test":"Use the gold gloss annotations that Phoenix14T and CSL-Daily already provide (the proposed method never uses them) to test temporal fidelity. For each training video, compute the optimal monotone alignment between the FPG-CTC pseudo-gloss sequence g_p and the gold gloss sequence g (e.g., dynamic time warping or Levenshtein constrained to preserve order), and report the percentage of pseudo-gloss tokens whose relative order matches a gold gloss at the aligned position, together with a chance baseline from shuffled g_p. If order agreement is high (>70%), the Section 3.1 premise is supported; if it is near chance or low (<40%), the CTC pretraining is being driven by translation-order labels rather than sign order.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SignLlama's headline gain depends on FPG-CTC pretraining: Table 14 shows BLEU-4 rises from 5.46 to 26.74 on Phoenix14T when FPG-CTC is added, so the central claim collapses if this pretraining is not doing what Section 3.1 says. Section 3.1 generates the pseudo-gloss sequence g_p by lemmatizing the target text, removing prepositions/conjunctions, and randomly dropping 20% of tokens, then applies CTC loss (Eq. 3) between video features and g_p. CTC is order-sensitive: it can insert blanks and repeats, but it cannot reorder labels. The premise is therefore that the filtered target-text tokens appear in the video in that same temporal order. For sign languages this is not guaranteed and often false: written translations use spoken-language word order, while signed productions have their own constituent order, which frequently diverges. The paper reports no human evaluation of pseudo-gloss fidelity; Appendix D's own CSLR WER of 78.8% on Phoenix14T shows the pseudo-glosses are weak recognition targets, with deletion rates around 73-75%, but WER alone does not measure order faithfulness. If the order premise fails, the CTC pretraining is aligning the visual encoder to translation-order labels rather than signed content, so the mechanism claimed for the large BLEU-4 gain is unsupported, and the benefit may be an artifact of imposing arbitrary temporal structure. This is the load-bearing concern; VPD and the comparison tables depend on the visual encoder produced by FPG-CTC.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SignLlama, a gloss-free sign language translation (GFSLT) model that couples a PoolFormer-based visual encoder with Llama 3.2 1B (or larger variants). Two ingredients are introduced: Filtered Pseudo-Gloss CTC Pretraining (FPG-CTC), which derives pseudo-gloss sequences from the target text via lemmatization, stopword removal, and 20% random dropping and then uses them as CTC supervision targets for the visual encoder; and Visual-Prioritized Distillation (VPD), a training scheme that runs a visual-textual prediction path and a visual-only prediction path, distilling the former into the latter to push the model to rely on visual features. Experiments on Phoenix14T, CSL-Daily, How2Sign, and OpenASL report BLEU-4 scores of 26.74, 24.76, 9.89 (test in Table 9; the text of Section 8 says 14.47, referring to the value in Appendix D Table 14), and 15.16 respectively. The authors claim competitive or state-of-the-art performance without using external sign language datasets or extra modalities.","tokens_in":19879,"tokens_out":4896,"duration_ms":45562,"significance":"If the proposed mechanisms work as described, the paper offers a simple and reproducible recipe for adapting LLMs to gloss-free sign language translation, an important and timely problem. The reported gains over strong baselines (e.g., a BLEU-4 jump from 5.46 to 26.74 on Phoenix14T when FPG-CTC is added) are substantial and could motivate further work on pseudo-gloss supervision and visual-prioritized training for multimodal LLMs. The paper also releases pretrained models across scales, which is a tangible community contribution. However, the significance hinges almost entirely on whether FPG-CTC actually aligns the visual encoder to the signed content in the video; if the pseudo-gloss order does not match sign order, the large empirical gains may arise from an unintended inductive bias rather than the claimed alignment mechanism.","major_comments":[{"comment":"The central premise of FPG-CTC is that the filtered target-text token sequence g_p appears in the video in the same temporal order as the signed content. CTC is alignment-based and order-sensitive: it can insert blanks and repetitions but cannot reorder labels. The paper provides no direct evidence that the lemmatized, stopword-filtered, randomly dropped text sequence actually corresponds to the order of signs in the video. For Phoenix14T and CSL-Daily, gold gloss annotations are available, so a quantitative comparison between pseudo-gloss order and gold-gloss order is feasible and should be reported (e.g., order-aware edit distance, or a per-sequence alignment error). The paper's own CSLR results in Table 14 show WERs of about 78% on Phoenix14T and 70% on CSL-Daily, with deletion rates of 73–75%; while high deletion rates do not directly prove order mismatch, they indicate that the pseudo-glosses are weak recognition targets, and the paper does not disentangle deletion from reordering. Without such evidence, the mechanism for the large FPG-CTC gain (5.46 to 26.74 BLEU-4 on Phoenix14T) is unsupported. This is a load-bearing concern because VPD and all comparison tables depend on the visual encoder produced by FPG-CTC.","section":"Section 3.1, Eq. (3); Appendix D, Table 14"},{"comment":"The 'Visual-Only Mask' used in the VPD visual-only prediction path is not specified precisely. The paper says 'the textual inputs are masked out using a Visual-Only Mask' but does not state whether text token embeddings are replaced by a special mask embedding, whether an attention mask is applied, or how the model is prevented from seeing previous ground-truth tokens when generating the full sequence from visual input alone. This matters because VPD's whole purpose is to create a path that relies exclusively on visual features; the exact masking mechanism determines what information is available at each position. Without a precise specification or pseudo-code, the method is not reproducible.","section":"Section 3.2, Eq. (5), Figure 1(b)"},{"comment":"Ablation results are reported as single runs without error bars, and hyperparameters λ (VPD distillation weight) and the FPG-CTC drop rate are tuned on the Phoenix14T validation set and then fixed across all other datasets. Several ablation comparisons differ by less than 1 BLEU-4 point (e.g., Table 2: L_vt-only 23.47 vs L_v-only 23.25; Table 4: Llama2 7B 26.26 vs Llama3.2 1B 26.74), so run-to-run variation could change the conclusions about which components are responsible for the improvement. The authors should report mean and standard deviation over at least three seeds for the main ablations and should justify the transfer of hyperparameters to CSL-Daily, How2Sign, and OpenASL, whose data distributions and sign languages differ substantially from Phoenix14T.","section":"Section 6, Tables 1–3; Appendix C, Tables 12–13"}],"minor_comments":[{"comment":"The text says 'improving the BLUE4 score' — 'BLUE4' should be 'BLEU-4'.","section":"Section 8, 'Evaluation on Phoenix14T Dataset'"},{"comment":"The table entry for Phoenix14T 'PG Voc in FPG-CTC' reads '30,01', which appears to be a typo or formatting error; please verify and fix all numeric entries in this table.","section":"Appendix A, Table 11"},{"comment":"The distillation term L_KD is defined as a KL divergence without any temperature parameter. If the standard softened knowledge distillation is intended, the temperature should be stated; otherwise, the formulation should clarify why no temperature is used.","section":"Section 3.2, Eq. (5)"},{"comment":"The sentence 'There are also some works focusing on designing pseudo gloss [9, 15]' cites references [9] and [15]; reference [9] (SignDINO) is a concurrent/related work from the same group and does not appear to use pseudo glosses, and [15] (MixSignGraph) is a graph-based approach. Please re-check the citation mapping.","section":"Section 2, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a high fraction of self-citations (SignGraph, MixSignGraph, SignDINO, SignPR, SLQA by the same group), and the presentation suggests incremental extension of the authors' own prior work. A revision should clarify the novel contributions relative to SignDINO (same first author) and MixSignGraph. The reviewer's main concern remains the unvalidated temporal-order premise of FPG-CTC; since gold glosses are available for two of the four datasets, the requested validation is straightforward and would substantially raise confidence in the reported results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. The recipe works: FPG-CTC pretraining plus VPD distillation lifts BLEU-4 from 5.46 to 26.74 on Phoenix14T, with similar jumps on three other datasets, and the ablations are clean enough that I believe the additive story. But there is a self-contradiction in the reported numbers: the How2Sign test BLEU-4 is 9.89 in Table 9 and 14.47 in both Section 8 and Appendix D Table 14. That cannot stand.\n\nWhat's new: the specific combination of filtered pseudo-gloss CTC pretraining for the visual encoder and a visual-only distillation path for the LLM. The components are known—pseudo-gloss pretraining and knowledge distillation—but the asymmetric teacher-student split for gloss-free SLT is not something I've seen. The paper also ships a lot of useful details: hyperparameter sweeps for lambda and drop rate, training and inference speed, and clear ablations.\n\nThe soft spots are in the mechanism, not the numbers. The stress test is right that the paper never checks whether the text-derived pseudo-glosses actually match the temporal order of signed content. CTC is order-sensitive, and the authors' own CSLR WER around 79% with 74% deletion rate suggests the pseudo-glosses are weak recognition targets. However, the gains don't collapse just because the mechanism is unproven. It is possible that any structured target helps the visual encoder produce more discriminative features. The authors can settle this in a day: gold glosses exist for Phoenix14T and CSL-Daily, so compare pseudo-gloss order to gloss order. Until they do, the claim that FPG-CTC aligns the encoder to signed content is unsupported. Their own Limitations section admits the pseudo-gloss generation is simple and leaves the question open.\n\nOther issues: no code or models released despite the contribution claim, no error bars anywhere, and the visual-only masking for VPD is unspecified—without that detail, reproduction is guesswork. The attention visualizations and t-SNE are suggestive, not evidence.\n\nThis paper is for people building LLM-based sign language translators. It deserves a serious referee, but my verdict would be 'revise': fix the inconsistency, release code/models or report enough detail to reproduce, and run the gold-gloss order check. The empirical contribution is strong enough to warrant that effort.","headline":"Solid empirical recipe for gloss-free sign language translation with LLMs, but the How2Sign numbers contradict each other and the core pretraining mechanism is untested, so the paper needs revision rather than rejection.","tokens_in":20449,"tokens_out":4342,"would_cite":false,"duration_ms":38194,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual-first training lifts gloss-free sign translation to 26.74","keywords":["sign language translation","gloss-free translation","large language models","pseudo-gloss supervision","CTC pretraining","visual-prioritized distillation","knowledge distillation","multimodal LLM fine-tuning"],"falsifier":"Retrain on Phoenix14T with the pseudo-gloss token order shuffled before FPG-CTC pretraining, keeping everything else fixed; if BLEU-4 stays near 26.74 instead of falling toward the 5.46 no-pretraining baseline, then the order and content fidelity of pseudo-glosses is not what carries the gain. A human check of whether the generated pseudo-glosses correspond to the signs actually performed in a random sample of videos would settle the same question directly.","tokens_in":1768,"feed_emoji":"🤟","tokens_out":2242,"duration_ms":103537,"temperature":0.7,"pith_summary":"This paper claims that a large language model can translate sign-language video into text without any gloss annotations, provided two specific problems are fixed: the mismatch between continuous video features and discrete text tokens, and the tendency of autoregressive models to lean on text tokens once both modalities are concatenated. The proposed SignLlama first pretrains its visual encoder with CTC supervision on pseudo-glosses—sequences built from the target sentence by lemmatization, removal of function words, and random dropping—so video features acquire a text-like, discrete organization. It then fine-tunes the language model with a visual-prioritized distillation in which a visual-only prediction path (text masked) is trained to imitate a standard visual-textual path, forcing the model to rely on the video rather than on already-generated text. On four public benchmarks the paper reports BLEU-4 scores of 26.74 on Phoenix14T, 24.76 on CSL-Daily, 9.89 on How2Sign, and 15.16 on OpenASL, with no external sign-language datasets and no extra modalities. If the claims hold, gloss-free translation stops being the weak sibling of gloss-supervised translation, and the main bottleneck moves from annotation cost to the fidelity of automatically generated pseudo-glosses.","feed_headline":"Visual-first training lifts gloss-free sign translation to 26.74","feed_subtitle":"Text-derived pseudo-glosses plus a visual-only distillation path get Llama to 26.74 BLEU-4 on Phoenix14T.","key_machinery":"The two carrying mechanisms are FPG-CTC and VPD. FPG-CTC is a pretraining scheme that converts the target sentence into a pseudo-gloss sequence—after lemmatization, removing prepositions and conjunctions, and randomly dropping a fixed fraction (0.2 in the final model) of words—and uses connectionist temporal classification loss to drive the visual encoder toward text-like discrete representations. VPD is a fine-tuning objective that runs two forward passes through the LLM: a teacher path with visual and textual tokens, and a student path with textual tokens masked, with cross-entropy on both paths plus a KL-divergence distillation term (weight $\\lambda=5$) pushing the student's distribution toward the teacher's; only the visual-only path is used at inference. Mechanism-wise, FPG-CTC manufactures the alignment signal that gloss labels normally provide, and VPD converts a text-conditioned language model into a video-conditioned one by making the text-free path the operative path.","core_discovery":"On the paper's own terms, the discovery is that the two obstacles to LLM-based gloss-free sign language translation are separable and each has a simple fix. FPG-CTC closes the visual-textual distribution gap: filtered pseudo-glosses generated from target text supervise the visual backbone through CTC loss, producing visual features that behave like discrete tokens and cluster near the text embedding distribution. VPD closes the visual-deprioritization gap: by distilling logits from the visual-textual path into a visual-only path, the model learns to generate the full target sentence from video alone, and attention maps shift from previous text tokens to visual tokens. The combination is what the paper credits for lifting Phoenix14T BLEU-4 from a 5.46 direct-fine-tuning baseline to 26.74, with consistent gains across four datasets; the authors also report that scaling the language model from 1B to 13B gives only marginal improvements at this range, suggesting the visual alignment stage carries most of the benefit.","pith_inferences":["Beyond the paper's experiments, the VPD recipe—distill a text-conditioned teacher into a vision-only student—is a general corrective for any multimodal LLM in which one modality dominates, since the gradient argument in Section 4 assumes only an autoregressive text-biased model.","The rule-based pseudo-gloss generator could be replaced by any text-derived sequence that preserves temporal order; on other action-to-language tasks without glosses, the same CTC pretraining would apply as an untested extension.","Not tested by the paper: the optimal random-drop rate probably tracks how much of the target sentence is left implicit in the signing, so datasets with more verbose text may need a higher drop rate than 0.2.","The reported 1B-to-13B plateau is a hint, not a proof, that visual-encoder quality and pseudo-gloss fidelity dominate LLM capacity in this task range; holding the encoder fixed and scaling only the LLM past 13B would test it."],"forward_implications":["Gloss-free translation can match or exceed many gloss-supervised baselines without any gloss annotation step, so gloss labels may not be necessary for high-quality sign language translation.","Text-only pseudo-gloss generation plus CTC is enough to discretize sign video features; the reported t-SNE visualizations show the visual features becoming more discriminative and text-like after FPG-CTC pretraining.","Visual-prioritized distillation reduces reliance on previous text tokens and addresses exposure bias; the reported attention maps show the model allocating more weight to visual tokens during generation.","Larger Llama sizes from 1B to 13B give only marginal gains in the paper's experiments, implying the bottleneck for this task range is visual alignment rather than raw language-model capacity.","The full pipeline needs only RGB video and paired text, with no external sign-language datasets or extra sensor modalities, which lowers the data requirements for deploying LLM-based sign translation."],"supporting_citations":[{"why":"Supplies the Phoenix14T benchmark used for the main ablations and final comparisons.","marker":"[2]"},{"why":"Establishes that gloss-label pretraining is critical for sign translation, the dependency FPG-CTC replaces with pseudo-glosses.","marker":"[4]"},{"why":"Is the strongest prior gloss-free baseline on How2Sign and OpenASL that SignLlama is measured against.","marker":"[5]"},{"why":"Is a recent Phoenix14T baseline whose BLEU-4 the paper claims to exceed by 1.28 points.","marker":"[9]"},{"why":"Supplies the pseudo-gloss idea and CTC-style visual-encoder pretraining that FPG-CTC simplifies and extends.","marker":"[15]"},{"why":"Is an LLM-based GFSLT baseline using codebooks, the alternative alignment strategy FPG-CTC is compared with.","marker":"[17]"},{"why":"Exemplifies the concatenated visual-textual autoregressive strategy that VPD is designed to counter and is a comparison baseline.","marker":"[23]"},{"why":"Proposes reweighting text tokens by visual relevance, the closest prior attempt at addressing visual deprioritization.","marker":"[45]"}],"fun_headline_variants":["Visual-first training boosts gloss-free sign translation to 26.74 BLEU","Filtered pseudo-gloss CTC plus visual distillation lifts sign translation to 26.74","SignLlama: Visual-prioritized distillation gets 26.74 BLEU-4 on Phoenix14T","Visual-first path with FPG-CTC pretraining lifts BLEU-4 to 26.74","Visual-prioritized distillation: key to 26.74 BLEU-4 in gloss-free SLT"],"cache_read_input_tokens":22528,"weakest_assumption_plain":"Everything rests on the assumption that the pseudo-gloss sequences—made by lemmatizing the target sentence, deleting function words, and randomly dropping one word in five—match, in order, what the signer actually performs; if that correspondence is wrong, the CTC pretraining aligns the video encoder to text-derived labels rather than to signed content, and the reported gains would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Visual-first training boosts gloss-free sign translation to 26.74 BLEU","Filtered pseudo-gloss CTC plus visual distillation lifts sign translation to 26.74","SignLlama: Visual-prioritized distillation gets 26.74 BLEU-4 on Phoenix14T","Visual-first path with FPG-CTC pretraining lifts BLEU-4 to 26.74","Visual-prioritized distillation: key to 26.74 BLEU-4 in gloss-free SLT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001104,"raw_usage":{"total_tokens":4658,"prompt_tokens":1055,"completion_tokens":3603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":3476}},"tokens_in":671,"tokens_out":3603,"duration_ms":23275,"temperature":1.0,"reasoning_tokens":3476,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:17:53.117646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain on Phoenix14T with the pseudo-gloss token order shuffled before FPG-CTC pretraining, keeping everything else fixed; if BLEU-4 stays near 26.74 instead of falling toward the 5.46 no-pretraining baseline, then the order and content fidelity of pseudo-glosses is not what carries the gain. A human check of whether the generated pseudo-glosses correspond to the signs actually performed in a random sample of videos would settle the same question directly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is a recent Phoenix14T baseline whose BLEU-4 the paper claims to exceed by 1.28 points."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the strongest prior gloss-free baseline on How2Sign and OpenASL that SignLlama is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Phoenix14T benchmark used for the main ablations and final comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes reweighting text tokens by visual relevance, the closest prior attempt at addressing visual deprioritization."}],"review_version":1}