{"id":"e4a9ff23-ad52-486e-a413-f861033d5f51","arxiv_id":"2412.16524","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLaVA-SLT, a three-stage large multimodal model with a hierarchical visual encoder and lightweight MLP connector, achieves state-of-the-art gloss-free sign language translation on CSL-Daily and Phoenix-2014T, approaching gloss-based accuracy when augmented with a new 401-hour dataset.","lead":"A new sign language translation model, LLaVA-SLT, combines a large language model with a hierarchical visual encoder trained in three stages to translate sign language video into spoken sentences without gloss annotations. It reports state-of-the-art results on two benchmarks and shows that adding a large self-collected video dataset narrows the gap with gloss-based methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The extra-data result in Table 3 is the sole support for the 'closes to gloss-based' claim, but CSL-400h is unreleased, GPT-4-scripted, and scenario-aligned with the CSL-Daily test set, so the gain cannot currently be separated from domain overlap or contamination.","rationale":"I agree with the reader's conditional assessment. The method is internally coherent: the three-stage pipeline is described in enough detail to reproduce its shape, the ablations in Tables 6-8 show that each proposed component contributes positively, and the Phoenix-2014T result in Table 4 provides a useful generalization check that does not depend on CSL-400h. These support the claim that the architecture works on standard in-distribution benchmarks. What I do not find established is the stronger headline claim that extra annotation-free data closes the gloss gap. The 25.23 BLEU-4 row is the only evidence for that claim, and it depends on an unreleased, scripted dataset whose text was generated by GPT-4 from predefined scenarios and whose scenario taxonomy overlaps CSL-Daily. This is a verification condition, not an accusation: releasing the data or the overlap analysis would settle it. The internal numerical mismatch between Table 3 (BLEU-4 20.42) and Table 6 row 4 (BLEU-4 20.92) for what appears to be the same 14B configuration adds minor uncertainty but is not the load-bearing issue. With the evidence as presented, the appropriate verdict remains conditional: credit the in-distribution method, but do not credit the scalability-to-gloss-free conclusion until CSL-400h and its relationship to the test set are independently audited.","tokens_in":22313,"tokens_out":6611,"duration_ms":60200,"concrete_test":"Release the CSL-400h text prompts (or a held-out split of them) and compute, for each CSL-Daily test reference, the maximum 4-gram overlap and BLEU-1 self-similarity against the CSL-400h prompt set after removing exact duplicates. Then report Table 3's BLEU-4 separately for test sentences whose best overlap is above versus below a threshold such as BLEU-1 > 0.6. If the extra-data gain concentrates in the high-overlap subset, the 'closing the gap' result is contaminated by scripted scenario overlap rather than supported by generalizable annotation-free scaling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical basis for the paper's central scalability claim is the last row of Table 3: adding CSL-400h raises BLEU-4 from 20.42 to 25.23, approaching the gloss-based TwoStream-SLT's 25.79. That row depends entirely on CSL-400h, a dataset that is not released and was constructed, per Supplement Sec. B, by having GPT-4 generate text prompts from predefined scenarios, mapping those prompts to dictionary-constrained gloss sequences, and then recording signers in a controlled lab with the same broad scenario categories that appear in CSL-Daily (work, campus, shopping, family, entertainment, medical, others). For the claim to hold, the gain must reflect generalizable sign-to-text knowledge, not the fact that the extra data is a synthetic, scripted distribution aligned with the test distribution. Because the text was generated first and the videos were produced to match it, the visual-to-text mapping in CSL-400h is artificially unambiguous; it is not a sample of naturally occurring signing. The paper reports no overlap analysis between CSL-400h prompts and CSL-Daily test references, no held-out signer or scenario evaluation, and no baseline trained on the same added data. The marginal gain therefore cannot currently be attributed to the method rather than to the construction of the extra data. The headline 'closes to gloss-based' is unverifiable as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLaVA-SLT, a large multimodal model framework for gloss-free sign language translation. The method is trained in three stages: (1) linguistic continued pretraining, in which a Qwen-2.5 LLM is further trained on a newly collected CSL-Corpus of sign-language-related text; (2) visual contrastive pretraining, in which a hierarchical DINOv2-based visual encoder with frame-, word-, and sentence-level features is aligned to an mBART text encoder using CLIP and SignCL losses; and (3) visual language tuning, in which a lightweight MLP connector maps word-level visual tokens into the LLM embedding space while the visual encoder and LLM are frozen, optionally followed by full tuning with LoRA. The model is evaluated on CSL-Daily and Phoenix-2014T. The paper reports state-of-the-art gloss-free results on both benchmarks and, when adding a newly collected 401-hour annotation-free dataset CSL-400h, claims that the model closely approaches gloss-based translation accuracy on CSL-Daily.","tokens_in":22654,"tokens_out":5315,"duration_ms":43702,"significance":"If the results hold, the paper makes a concrete contribution: it demonstrates that a scaled LLM with a visually pretrained hierarchical encoder can reduce the gap between gloss-free and gloss-based sign language translation, and it provides a systematic ablation of LLM scale, visual encoder architecture, input resolution, data scaling, connector design, and prompting. The ablations are unusually thorough for this problem and give useful evidence about which components matter. The main limitation, as discussed below, is that the headline 'closes to gloss-based accuracy' result depends on an unreleased, internally constructed dataset whose distributional relationship to the test set is not analyzed; additionally, there is an unresolved numerical inconsistency in the reported BLEU-4 scores for the same model. The paper does not provide code or data links despite a statement in the broader-impact section about open-sourcing, so the central extra-data claim is not currently reproducible.","major_comments":[{"comment":"There is an internal inconsistency in the reported BLEU-4 for the same model. Table 3 lists LLaVA-SLT (ours) with BLEU-1 52.15, BLEU-4 20.42, and ROUGE 51.26, while Table 6 row 4 reports the same configuration (Qwen-2.5 14B after continued pretraining) with BLEU-1 52.15, BLEU-4 20.92, and ROUGE 51.26. Because the BLEU-1 and ROUGE values are identical, these rows evidently refer to the same model, so the BLEU-4 discrepancy is either a typo or a difference in evaluation protocol. This must be resolved, since the headline comparison against GFSLT-VLP-SignCL (16.16 vs. 20.42) and the LLM-scaling trend in Table 6 depend on which number is correct.","section":"Table 3 and Table 6"},{"comment":"The abstract's claim that the method 'closes to the gloss-based accuracy' rests on the last row of Table 3, which adds CSL-400h data and raises BLEU-4 from 20.42 to 25.23. However, CSL-400h is not released, and its construction, described in Supplement Sec. B, involves GPT-4-generated text prompts from predefined scenario categories (work, campus, shopping, family, entertainment, medical, others) that are the same broad categories as in CSL-Daily, with videos recorded in a controlled lab setting. The paper reports no overlap analysis between CSL-400h prompts and CSL-Daily test references, no held-out signer or scenario evaluation, and no baseline trained on the same added data. As presented, the marginal gain cannot be attributed to the method's scalability rather than to distributional alignment between the extra training data and the test set. To support the claim, the authors should either release the dataset, or provide experiments that control for domain overlap, such as a signer- and scenario-split evaluation, a comparison baseline using the same extra data with an existing method, or an analysis of prompt/test overlap. Without such evidence, the 'closes to gloss-based accuracy' claim should be substantially softened or removed.","section":"Table 3 and Supplement Sec. B"},{"comment":"The data-scaling ablation in Table 7 rows 4-5 is not fully specified. The text says the training data is expanded from CSL-Daily to 200 and 400 hours using CSL-400h, but it does not state whether these models also use CSL-400h during visual language tuning, how the '50% extra data' subset is sampled, or how the extra-data rows in Table 7 relate to the 'w/ extra data' row in Table 3. This ambiguity matters because the final translation metric aggregates the effects of visual pretraining and tuning. Please state the exact training protocol for each configuration.","section":"Table 7 and Section 4.4"}],"minor_comments":[{"comment":"The phrase 'closes to the gloss-based accuracy' is grammatically awkward and imprecise; I suggest 'closes the gap to gloss-based methods' or 'approaches gloss-based accuracy'.","section":"Abstract and Section 4.2"},{"comment":"The caption says 'with the same training set (ours)' but this is ambiguous; please specify explicitly that 'ours' in the row without extra data is trained only on the CSL-Daily training split.","section":"Table 3 caption"},{"comment":"Reference [48] lists 'LoRA: Low-rank adaptation of large language models' with an incomplete year '202.'; please correct it.","section":"References"},{"comment":"The relationship between the local attention window size and the nearest-neighbor downsampling step (step 4) is not explained; please define how a 'word' is determined for the word-level visual tokens.","section":"Section 3.2"},{"comment":"The phrase 'tamed for sign language translation' should be 'tuned for sign language translation' to match the terminology used in the rest of the paper.","section":"Figure 1 caption"},{"comment":"The sentence 'it even closes to the gloss-based accuracy' appears in prose as well; also 'surpasses gloss-based methods in BLEU-1 scores' should explicitly say 'BLEU-1 score on the CSL-Daily test set' for precision.","section":"Section 4.2"},{"comment":"The limitation about 'Real-world Scenarios' is stated only in the supplement; consider adding a sentence in the main conclusion to acknowledge that the reported results are on lab-recorded data.","section":"Supplement Sec. D"}],"recommendation":"major_revision","confidential_remarks":"The core technical direction is interesting and the ablation study is a strength, but the central 'closes to gloss-based' claim depends on a dataset whose provenance and release status are not established. The BLEU-4 inconsistency between Tables 3 and 6 is concerning and should be checked by the authors before resubmission. I would also encourage the editor to ask whether the authors can release at least a sample or a detailed protocol of CSL-400h; without that, the extra-data result cannot be independently verified. The novelty of the method itself is moderate, as it combines existing contrastive pretraining, LoRA, and LLaVA-style tuning, but the scaling study and the word-level visual token idea are worth publishing if the numerical issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee. The core method is coherent: linguistic continued pretraining on a sign-language corpus, a hierarchical visual encoder with local-attention word tokens, CLIP-style outer contrastive loss plus SignCL inner regularization, and a frozen-LLM MLP connector. The ablations are genuine and informative. The local-attention word-level token ablation (BLEU-4 19.88 to 14.87 when removed) and the text-encoder swap (Qwen supervision dropping to 13.42) both show the design choices matter. The Phoenix-2014T result without continued pretraining is also a useful generalization data point. The numerical claim on CSL-Daily without extra data (BLEU-4 20.42 vs 16.16 for GFSLT-VLP-SignCL) is a real advance for gloss-free methods, and the gains from scaling the LLM from 3B to 14B are consistent. The soft spots are real but not fatal. First, the internal inconsistency: Table 3 reports BLEU-4 20.42 for the 14B model while Table 6 Row 4 reports 20.92 for the same configuration. That must be reconciled. Second, the central scalability claim depends on CSL-400h: 264,461 videos, unreleased, GPT-4-scripted text, recorded in a lab with the same scenario categories as CSL-Daily, and no baseline trained on the same added data. The paper says the dataset is annotation-free, which is true in the gloss sense, but the text prompts were generated first and videos were produced to match, so the visual-to-text mapping is artificially clean. The gain from 20.42 to 25.23 cannot currently be attributed to the method rather than to dataset construction. The paper does not report overlap analysis, signer-split evaluation, or a comparison baseline with the same extra data. The limitation section does acknowledge controlled settings and short-term context, which is honest, but it does not address the more specific concern of scenario alignment between CSL-400h and CSL-Daily. Third, the abstract's 'outperforms state-of-the-art' is too broad as written; the Phoenix result is a narrow improvement and the CSL comparison without extra data is the fair one. The citation pattern looks fine; SignCL, GFSLT-VLP, and Sign2GPT are all cited and the differences are described accurately. The paper ships no code or data, which hurts reproducibility but is not unusual for an arXiv preprint at this stage. If I were the editor, I would send this to review. The method section is solid, the ablations support the main claims, and the extra-data question is exactly what a good referee can sharpen. The authors need to release CSL-400h or at least provide a signer-disjoint evaluation and a same-data baseline before the headline claim is credible. Recommendation: accept for peer review with the expectation of heavy revision on the dataset-confound issue.","headline":"A technically solid gloss-free SLT pipeline with honest ablations, but the headline 'closes to gloss-based' rests on an unreleased scripted dataset whose gain cannot be separated from test-domain overlap.","tokens_in":23215,"tokens_out":718,"would_cite":true,"duration_ms":27849,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage multimodal recipe brings gloss-free sign language translation to the level of gloss-supervised models.","keywords":["sign language translation","gloss-free","large multimodal model","visual language","contrastive pretraining","hierarchical visual encoder","CSL-400h","LLM scaling"],"falsifier":"Train LLaVA-SLT on CSL-400h and evaluate on a sign language test set recorded in natural, uncontrolled conditions (for example, conversational or broadcast sign video with human-written references); if the BLEU-4 advantage over the lab-only baseline shrinks substantially, the gap-closing result is an artifact of matching lab conditions. Alternatively, an external audit checking CSL-400h for near-duplicate videos or lexical overlap with the CSL-Daily test set could settle whether the extra-data gain is genuine.","tokens_in":22089,"feed_emoji":"🧏","tokens_out":7415,"duration_ms":57871,"temperature":0.7,"pith_summary":"Sign language translation has long depended on gloss annotations—word-by-word labels of signs—that are costly and slow to produce. The paper argues that a large multimodal model, adapted to sign language with a three-stage training scheme, can translate from raw video without glosses, and that adding annotation-free video data closes most of the gap with gloss-based systems. On the CSL-Daily benchmark, the model reaches a BLEU-4 of 20.42 with the standard training set, beating prior gloss-free approaches, and 25.23 when an additional 401 hours of unlabeled-style video is added, close to the 25.79 of a strong gloss-supervised baseline. The result matters because it points to a scalable path for sign language translation: collecting raw video with text descriptions rather than paying experts to gloss it.","feed_headline":"Unlabeled video closes the gloss-free sign-language gap","feed_subtitle":"401 hours of annotation-free video plus a 14B LLM match gloss-supervised BLEU-4 on CSL-Daily.","key_machinery":"The load-bearing mechanism is the hierarchical visual encoder: a frozen DINOv2 backbone (finetuned with LoRA) provides frame-level features; a local-attention transformer pools them into isolated word-level tokens; and a full-attention transformer encodes the token sequence into a sentence-level query. This visual \"language\" is aligned to a pretrained multilingual text encoder (mBART) via an outer CLIP-style contrastive loss plus an inner SignCL loss, and then mapped into the language-model token embedding space by a two-layer MLP connector while the language model and visual encoder are frozen. The response prompting—a task prompt and a format prompt—constrains the output space of the language model, which the ablations show is important for translation accuracy.","core_discovery":"The central claim is that a three-stage \"trilogy\"—continued pretraining of a scaled-up language model on sign-language textual corpora, contrastive pretraining of a hierarchical visual encoder against a frozen text encoder, and end-to-end visual language tuning with a lightweight MLP connector—produces word-level visual tokens that behave like a \"visual language\" compatible with language-model token embeddings. The paper demonstrates that this design, LLaVA-SLT, outperforms previous gloss-free methods on CSL-Daily and Phoenix-2014T, and that when training data is expanded with the annotation-free CSL-400h dataset, its BLEU-4 approaches the gloss-based TwoStream-SLT model. The authors interpret this as evidence that gloss-free translation can be scaled with annotation-free multimodal data, potentially replacing labor-intensive glossing.","pith_inferences":["If the extra-data gains come from the distributional similarity of CSL-400h and CSL-Daily—both recorded in controlled laboratories—the benefit may shrink on real-world footage; an independent natural test set would settle this.","The text side of CSL-400h was generated by a large language model, so the dataset may encode a canonical, well-formed style of Chinese that inflates BLEU scores against textbook references; training on naturally occurring sign language text could yield lower but more realistic numbers.","A direct test of the framework's scaling claim would be to apply it to open-domain sign video with automatic captions and measure whether BLEU continues to rise with hundreds of additional hours of unlabeled data.","The reliance on a frozen visual backbone (DINOv2) means the \"visual language\" is built on features not optimized for sign; an end-to-end pretraining of the whole encoder might remove that constraint, though this is not explored in the paper."],"forward_implications":["If the result holds, sign language translation research can shift from expensive gloss annotation to collecting raw video paired with text, which is available at web scale.","Scaling the base language model from 3B to 14B and scaling video data from 20 to 400 hours both improve BLEU scores, suggesting a predictable scaling path for the approach.","Word-level visual tokens outperform sentence-level features as inputs to the language model, giving a concrete design rule for future sign-language multimodal models.","The model transfers to German sign language without continued linguistic pretraining, so the visual component of the recipe is language-agnostic.","Higher input resolution (336 squared) and a larger visual backbone materially improve accuracy, implying that fine-grained hand and face cues are the dominant signal."],"supporting_citations":[{"why":"Supplies the inner contrastive loss (SignCL) used to regularize visual embeddings and provides the strongest prior gloss-free baseline that the paper must beat.","marker":"[105]"},{"why":"Establishes the gloss-free visual-language pretraining paradigm that the paper builds on and is the direct source of the contrastive pretraining baseline.","marker":"[116]"},{"why":"A prior LLM-based gloss-free method using hand-crafted pseudo-glosses; the paper compares its word-level alignment strategy against this approach.","marker":"[100]"},{"why":"An LLM-based gloss-free method that supervises the visual encoder with text generation; serves as a comparison point for translation quality.","marker":"[25]"},{"why":"The strongest gloss-based model (TwoStream-SLT) whose accuracy LLaVA-SLT approaches when extra annotation-free data is added.","marker":"[24]"},{"why":"Provides the CSL-Daily benchmark, the main evaluation dataset that defines the gloss-free versus gloss-based comparison.","marker":"[118]"},{"why":"Provides the Phoenix-2014T German sign language dataset used to demonstrate cross-lingual generalization.","marker":"[17]"},{"why":"The frozen visual backbone (DINOv2) that the hierarchical encoder builds on, providing frame-level features.","marker":"[81]"},{"why":"The pretrained multilingual text encoder (mBART) that supervises the visual contrastive pretraining.","marker":"[75]"},{"why":"The visual instruction tuning paradigm and MLP connector design that the paper adapts for sign language translation.","marker":"[73]"}],"fun_headline_variants":["Visual language tuning lifts gloss-free sign translation","LLaVA-SLT: three-stage tuning matches gloss-based SLT","Annotation-free videos narrow gloss-free gap in SLT","Word-level visual embeddings boost sign translation","LLaVA-SLT uses unlabeled data to rival gloss supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline claim that annotation-free data closes the gap with gloss-based models rests on the newly collected CSL-400h dataset—264,461 videos recorded in a controlled lab with text generated by a large language model—which is assumed to be a representative, unbiased sample of Chinese Sign Language; the dataset is not publicly released and its collection protocol is not independently audited.","fun_headline_variants_meta":{"raw":{"variants":["Visual language tuning lifts gloss-free sign translation","LLaVA-SLT: three-stage tuning matches gloss-based SLT","Annotation-free videos narrow gloss-free gap in SLT","Word-level visual embeddings boost sign translation","LLaVA-SLT uses unlabeled data to rival gloss supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1553,"prompt_tokens":957,"completion_tokens":596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":573,"tokens_out":596,"duration_ms":5289,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:29:20.552378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LLaVA-SLT on CSL-400h and evaluate on a sign language test set recorded in natural, uncontrolled conditions (for example, conversational or broadcast sign video with human-written references); if the BLEU-4 advantage over the lab-only baseline shrinks substantially, the gap-closing result is an artifact of matching lab conditions. Alternatively, an external audit checking CSL-400h for near-duplicate videos or lexical overlap with the CSL-Daily test set could settle whether the extra-data gain is genuine.","supporting_citations":[{"cited_title":"Sign language translation with iterative prototype","cited_arxiv_id":null,"evidence_quote":"Supplies the inner contrastive loss (SignCL) used to regularize visual embeddings and provides the strongest prior gloss-free baseline that the paper must beat."},{"cited_title":"Gloss-free sign language translation: Improving from visual-language pretraining","cited_arxiv_id":null,"evidence_quote":"Establishes the gloss-free visual-language pretraining paradigm that the paper builds on and is the direct source of the contrastive pretraining baseline."},{"cited_title":"Improv- ing sign language translation with monolingual data by sign back-translation","cited_arxiv_id":null,"evidence_quote":"Provides the CSL-Daily benchmark, the main evaluation dataset that defines the gloss-free versus gloss-based comparison."},{"cited_title":"Visual alignment constraint for continuous sign language recognition","cited_arxiv_id":null,"evidence_quote":"The frozen visual backbone (DINOv2) that the hierarchical encoder builds on, providing frame-level features."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"The pretrained multilingual text encoder (mBART) that supervises the visual contrastive pretraining."}],"review_version":1}