{"id":"2efa5d33-d27b-4425-96ca-d7a3711b9839","arxiv_id":"2411.16789","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A gloss-free sign language translation framework, MMSLT, augments sign video features with MLLM-generated textual descriptions and achieves state-of-the-art BLEU-4 scores on PHOENIX14T and CSL-Daily.","lead":"The authors build a sign language translator that first asks a multimodal AI model to write text descriptions of the signer's hands, face, and movements, then combines those descriptions with the video to produce spoken-language sentences. It reports the best published gloss-free translation scores on the PHOENIX14T and CSL-Daily benchmarks, which could reduce reliance on expensive gloss annotations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set selection in §5.3 biases the reported SOTA: all ablations and the final configuration are chosen on the PHOENIX14T test set, so the +2.33 BLEU-4 gain over SignLLM needs a dev-selected rerun to stand.","rationale":"Central claim: MMSLT is SOTA gloss-free SLT, demonstrated by test BLEU-4 gains. For this claim to hold, the test numbers must reflect generalization, not choices made by peeking at test. That condition is violated in §5.3. The paper is otherwise coherent: the MMLP pre-training loss (Eq. 6) combines contrastive alignment (Eq. 5) and a mapper MSE (Eq. 2), and the ablation pattern is plausible. I credit the authors for releasing code and for consistent gains across tables. But because every design decision—which MLLM, which prompt, which description encoder, whether to include each module—is evaluated on test, the reported SOTA is a selected statistic. The PHOENIX14T dev/test gap asymmetry (+0.22 on dev vs +2.33 on test) is a red flag, though not proof by itself. A dev-only selection rerun is the decisive check. If it fails, the paper's central empirical claim is unsupported; if it passes, the claim is much stronger. This does not change the reader's CONDITIONAL verdict, but it sharpens the condition: the required revision is not optional polish, it is a re-selection experiment. The reader's stated weakest assumption (description mapper) is a mechanism-level concern; I see it as secondary because even a perfect mapper would not help if the SOTA number is an artifact. Conversely, even an imperfect mapper could still leave the empirical SOTA true. Hence partial agreement.","tokens_in":17551,"tokens_out":4394,"duration_ms":194097,"concrete_test":"Use the released code to rerun PHOENIX14T with model selection restricted to dev: pick MLLM, prompt, description encoder, λ, and LoRA rank using dev BLEU-4 only, then evaluate the chosen configuration once on test, repeating with 3 seeds. If the dev-selected test BLEU-4 remains above SignLLM's 23.40, the SOTA claim survives; if it drops below, the reported advantage is an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support \"MMSLT is SOTA gloss-free SLT,\" the reported test numbers must be unbiased estimates. Section 5.3 opens \"we conduct all ablation experiments using the test set of PHOENIX14T,\" so the choices of MLLM (Table 4), prompt (Table 5), description encoder (Table 6), and module combination (Table 3) are all made by maximizing test BLEU-4. The final system is therefore selected on the same test set that is later reported as SOTA. This is not a mere reporting nit: on PHOENIX14T dev, MMSLT's BLEU-4 is 25.47 versus SignLLM's 25.25 (+0.22), while on test it is 25.73 versus 23.40 (+2.33); a much larger gap on the set used for selection is exactly the signature of selection bias. With no error bars or multiple seeds reported, the headline margin could shrink or vanish under honest model selection. The description-mapper mechanism is a separate concern, but it is secondary: even if the mapper is imperfect, a fair dev-selected comparison would still reveal whether the method actually beats SignLLM. The test-set selection directly threatens the central empirical claim, so it is the load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMSLT, a gloss-free sign language translation framework that uses an off-the-shelf multimodal large language model (MLLM) to generate textual descriptions of sign language components from video frames. These descriptions are encoded with a frozen BERT and fused with ResNet visual features via a modality adapter; the fused features are aligned with spoken sentences through contrastive learning and passed to an mBART decoder for translation. A description mapper (a two-layer MLP) is trained to predict the description embeddings from visual features so that the MLLM is not needed at inference. Experiments on PHOENIX14T and CSL-Daily report state-of-the-art gloss-free results, e.g., +2.33 BLEU-4 over SignLLM on PHOENIX14T test and +5.36 on CSL-Daily test. Ablations in Sec. 5.3 show that each component contributes to the final performance.","tokens_in":17866,"tokens_out":2584,"duration_ms":62951,"significance":"If the reported gains hold under a sound evaluation protocol, the paper makes a meaningful contribution: it is the first to show that off-the-shelf MLLM-generated descriptions can be leveraged, without extra gloss supervision, to improve gloss-free SLT, and it provides a computationally efficient inference scheme via the description mapper. The method is clearly described, the code is released, and the ablations (Tables 3–6) are informative. The main risk is that the central 'state-of-the-art' claim rests on model choices made using the test set itself, which can substantially inflate the reported margins. A dev-selected re-run is needed to establish the actual gain.","major_comments":[{"comment":"","section":"Sec. 5.3, Tables 3–6"},{"comment":"","section":"Sec. 4.2, Eq. (1)–(2); Table 3, rows (5)–(6)"}],"minor_comments":[{"comment":"","section":"Table 6"},{"comment":"","section":"Table 4"},{"comment":"","section":"Eq. (7) and Sec. 4.3"},{"comment":"","section":"Fig. 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The main protocol flaw is that all model selection and ablations are performed on the test set (Sec. 5.3). This is a serious issue that undermines the headline SOTA claim; a dev-selected re-run is necessary before the paper can be accepted. The description-mapper concern is real but secondary. I would not reject the paper outright: the method is novel, clearly presented, and the released code will help the community. However, the revision must include a fair selection protocol and dev/test reporting for the final configuration. If the dev-selected margin over SignLLM turns out to be small (as the current dev numbers suggest), the authors should temper the SOTA claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is genuinely new and worth engaging with — using an off-the-shelf MLLM to verbalize sign components, then training a small mapper so the MLLM isn't needed at inference. That's a sensible way to inject textual supervision without a gloss label. The paper also ships code, which is more than many in this area. And the ablations, even if flawed in selection, suggest each piece (descriptions, alignment, mapper) contributes.\n\nThe soft spot is exactly where the reader's stress test lands: Section 5.3 openly says all ablations use the PHOENIX14T test set. That includes the choice of MLLM (Table 4), prompt (Table 5), description encoder (Table 6), and module combination (Table 3). So the final configuration is selected on the same test set that is reported as SOTA. On PHOENIX14T dev, MMSLT beats SignLLM by only +0.22 BLEU-4; on test it's +2.33. That jump is the signature of selection bias, not evidence of a better model. With no error bars or multiple seeds, the headline 'SOTA' claim is fragile.\n\nI also think the description mapper deserves a sharper look. The mapper predicts BERT embeddings from ResNet features, and at inference the MLLM text is gone. If the predicted embeddings don't retain the information that made the text useful, the benefit at deployment could evaporate. The paper's own Table 3 row (5) vs (6) shows DM adds 0.69 BLEU-4 on the test set; that's a modest gain and it's from the same selected set. Nothing here makes me think the method collapses, but the claim that the descriptions are the engine of improvement isn't yet nailed down.\n\nThe citation pattern is fine; they cover the relevant gloss-free prior work (GFSLT-VLP, SignLLM, Sign2GPT, FLa-LLM) and don't overclaim novelty relative to it. The qualitative examples look plausible.\n\nBottom line: this is a serious paper with a real idea, currently overclaimed. A referee should ask for (1) re-running ablations and model selection on dev, then reporting test once, (2) multiple seeds or error bars, and (3) an analysis of what the description mapper actually encodes vs. what the visual encoder already knows. With those changes, I'd be comfortable with the SOTA claim. I'd send it to review rather than desk reject.","headline":"Novel and plausible method for gloss-free SLT via MLLM descriptions, but the headline SOTA numbers are compromised by test-set selection in the ablations; a dev-selected rerun is needed before trusting the +2.33/+5.36 claims.","tokens_in":18410,"tokens_out":2078,"would_cite":true,"duration_ms":20109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMSLT shows that MLLM-written descriptions of sign language components, fused with video features during pre-training, set a new state of the art for gloss-free sign language translation on PHOENIX14T and CSL-Daily.","keywords":["sign language translation","gloss-free","multimodal large language models","visual-language pre-training","description mapper","PHOENIX14T","CSL-Daily","contrastive alignment"],"falsifier":"Run MMSLT's full training pipeline, but at inference replace the mapper's predicted description embeddings with the true BERT embeddings of the MLLM descriptions (i.e., use the MLLM at inference). If BLEU-4 does not improve over the mapper version, or if replacing the predicted embeddings with random vectors of the same shape preserves the reported BLEU-4, then the contribution attributed to the semantic content of the descriptions would be called into question.","tokens_in":17380,"feed_emoji":"🧏","tokens_out":4129,"duration_ms":36108,"temperature":0.7,"pith_summary":"The paper tries to establish that gloss-free sign language translation improves when the model sees both the sign video and a text description of the sign components (hand shape, gaze, mouth) generated by an off-the-shelf multimodal large language model. To keep inference cheap, a small description mapper learns to predict the embeddings of those descriptions from visual features alone, so the MLLM is only needed during pre-training. On PHOENIX14T, the method reports a test BLEU-4 of 25.73, beating the previous gloss-free state of the art by +2.33, and on CSL-Daily it reports 21.11, a +5.36 gain. The central claim is that the two modalities complement each other and that aligning the fused representation with spoken sentences bridges the modality gap better than using vision alone.","feed_headline":"Sign AI translates better after reading MLLM descriptions of gestures","feed_subtitle":"Adding text captions of hand and face cues lifts BLEU-4 to 25.73 on PHOENIX14T and 21.11 on CSL-Daily.","key_machinery":"The central mechanism is the multimodal-language pre-training module (MMLP), which contains four components: a description mapper (a two-layer MLP) that predicts the BERT [CLS] embedding of an MLLM-generated sign description from ResNet18 visual features; a modality adapter consisting of 1D convolution, max-pooling, and an MLP that fuses visual and predicted description features; a 12-layer mBART encoder fine-tuned with LoRA that extracts long-term multimodal features; and a contrastive alignment loss (with a learnable temperature) that pulls pooled multimodal features toward pooled spoken-sentence features from a frozen mBART encoder. The MLLM itself is used only in the GSD-MLLM pre-processing step, where a prompt focused on hands and face is fed to LLaVA-OneVision 7B to generate per-frame descriptions.","core_discovery":"The paper claims that MMSLT, a gloss-free sign language translation framework, outperforms prior gloss-free methods by leveraging multimodal large language models to generate detailed textual descriptions of sign language components from video frames. During pre-training, the description features are fused with visual features through a modality adapter and aligned with spoken sentences via contrastive learning, and a description mapper predicts the description embeddings from visual features so that the MLLM is not needed at inference. On the PHOENIX14T and CSL-Daily benchmarks, the approach achieves state-of-the-art gloss-free BLEU-4 scores of 25.73 and 21.11 respectively, outperforming SignLLM by +2.33 and +5.36, and also improving ROUGE scores, indicating better handling of long phrases and context.","pith_inferences":["Since the description mapper is trained to predict BERT embeddings of the MLLM descriptions from visual features, the benefit at deployment ultimately depends on how much of the description content is already recoverable from the visual features; if the mapper mostly recovers information already present in the ResNet features, the text modality may be adding less than the pre-training alignment it","A natural extension would be to distill the MLLM directly into the mapper using the full description texts (or their token embeddings) as supervision, rather than only the sentence-level [CLS] embedding, which could retain more fine-grained information.","The same recipe could transfer to other video-to-language tasks, such as instructional video captioning or action recognition, where an MLLM can provide auxiliary textual descriptions of fine-grained motion without requiring manual annotation.","The reported gains could be tested for robustness by evaluating on a low-resource sign language dataset or on signers unseen during training, where the diversity of MLLM descriptions might have a larger or smaller effect."],"forward_implications":["Gloss-free sign language translation can be improved without any gloss annotations by adding MLLM-generated text descriptions as a second input modality during pre-training.","Because the description mapper approximates the description embeddings at inference, the computational cost of the MLLM is avoided at deployment, making the approach practical for real-time use.","The method generalizes across two languages and datasets (German PHOENIX14T and Chinese CSL-Daily) and across multiple MLLMs, with most tested MLLMs outperforming the previous gloss-free state of the art.","The choice of prompt matters: a prompt that explicitly asks for motion and gestures of the person focused on hands and face yields better descriptions and higher translation quality than simpler or more generic prompts."],"supporting_citations":[{"why":"SignLLM is the previous gloss-free state-of-the-art baseline that MMSLT outperforms on both PHOENIX14T and CSL-Daily.","marker":"[18]"},{"why":"GFSLT-VLP provides the visual-language pre-training framework (aligning sign video features with spoken sentences) that MMSLT extends to multimodal input.","marker":"[67]"},{"why":"LLaVA-OneVision is the image-based MLLM used in GSD-MLLM to generate the sign language descriptions from video frames.","marker":"[29]"},{"why":"PHOENIX14T is the German sign language benchmark dataset used for evaluation.","marker":"[5]"},{"why":"CSL-Daily is the large-scale Chinese sign language benchmark dataset used for evaluation.","marker":"[68]"},{"why":"BERT is the frozen description encoder that produces the sentence-level embeddings the description mapper is trained to predict.","marker":"[15]"},{"why":"mBART provides the multimodal encoder and the translation decoder backbones, initialized from pre-trained parameters.","marker":"[48]"},{"why":"LoRA is used for parameter-efficient fine-tuning of the multimodal encoder while preserving pre-trained knowledge.","marker":"[22]"}],"fun_headline_variants":["MLLM descriptions of signs lift gloss-free translation to SOTA","Sign language translation improved by MLLM-generated text cues","Gloss-free SLT: fusing MLLM text with video boosts BLEU-4","MLLMs help sign translation by describing hand and face cues","MMSLT: using MLLM text to align sign video with spoken sentences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The two-layer description mapper is assumed to predict the BERT embeddings of the MLLM descriptions from visual features closely enough that the model at inference retains the translation benefit of the descriptions, even though the MLLM itself is no longer present.","fun_headline_variants_meta":{"raw":{"variants":["MLLM descriptions of signs lift gloss-free translation to SOTA","Sign language translation improved by MLLM-generated text cues","Gloss-free SLT: fusing MLLM text with video boosts BLEU-4","MLLMs help sign translation by describing hand and face cues","MMSLT: using MLLM text to align sign video with spoken sentences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3702,"prompt_tokens":894,"completion_tokens":2808,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2710}},"tokens_in":510,"tokens_out":2808,"duration_ms":19438,"temperature":1.0,"reasoning_tokens":2710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:22:35.407953+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MMSLT's full training pipeline, but at inference replace the mapper's predicted description embeddings with the true BERT embeddings of the MLLM descriptions (i.e., use the MLLM at inference). If BLEU-4 does not improve over the mapper version, or if replacing the predicted embeddings with random vectors of the same shape preserves the reported BLEU-4, then the contribution attributed to the semantic content of the descriptions would be called into question.","supporting_citations":[{"cited_title":"Llms are good sign language translators","cited_arxiv_id":null,"evidence_quote":"SignLLM is the previous gloss-free state-of-the-art baseline that MMSLT outperforms on both PHOENIX14T and CSL-Daily."},{"cited_title":"Gloss-free sign language translation: Improving from visual- language pretraining","cited_arxiv_id":null,"evidence_quote":"GFSLT-VLP provides the visual-language pre-training framework (aligning sign video features with spoken sentences) that MMSLT extends to multimodal input."},{"cited_title":"Neural sign language trans- lation","cited_arxiv_id":null,"evidence_quote":"PHOENIX14T is the German sign language benchmark dataset used for evaluation."},{"cited_title":"Improving sign language translation with monolingual data by sign back-translation","cited_arxiv_id":null,"evidence_quote":"CSL-Daily is the large-scale Chinese sign language benchmark dataset used for evaluation."},{"cited_title":"Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019","cited_arxiv_id":null,"evidence_quote":"BERT is the frozen description encoder that produces the sentence-level embeddings the description mapper is trained to predict."},{"cited_title":"Mul- tilingual translation with extensible multilingual pretraining and finetuning, 2020","cited_arxiv_id":null,"evidence_quote":"mBART provides the multimodal encoder and the translation decoder backbones, initialized from pre-trained parameters."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA is used for parameter-efficient fine-tuning of the multimodal encoder while preserving pre-trained knowledge."}],"review_version":1}