{"id":"d150fccd-449b-4d19-aa7f-7bce6d93408b","arxiv_id":"2507.09266","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SAGE uses a frozen sign-segmentation model to turn sign videos into about half as many visual tokens as prior methods, then aligns those tokens with a language model to reach BLEU-4 of 24.10 on PHOENIX14T.","lead":"This paper proposes SAGE, a sign language translation system that cuts each video into a few meaningful chunks, called segments, and translates those chunks into text. It reports better translation scores than prior gloss-free systems on a standard German Sign Language benchmark while using fewer tokens and less memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an undisclosed frozen segmentor [25]: if Hands-On was trained on PHOENIX14T or its boundaries are noisy, SAGE's BLEU-4 gain and token-efficiency are not an independent test; boundary quality and training-data overlap must be shown.","rationale":"The Reader identified the same load-bearing assumption: the frozen Hands-On segmentor must be accurate and its training data must not overlap PHOENIX14T. I agree with that assessment. I also considered the missing error bars, the undisclosed retrained GFSLT protocol, and the GPU-memory comparison across different batch sizes and hardware, but those are secondary to the segmentor question because they should be fixable with additional reporting. If the segmentor is clean and produces semantically valid boundaries, the method is plausible: the ablations in Tables 4 and 5 support the CLCL objective and dual-level supervision, and a code release is promised. If the segmentor is contaminated by PHOENIX14T or produces noisy boundaries, the central SOTA and token-efficiency claims collapse, regardless of how well the rest of the pipeline is engineered. The paper's reference to supplementary materials in Section 3.1.1 is not sufficient, since no supplementary material is present in the submitted text. Therefore, the concrete check is to measure boundary quality against the available PHOENIX14T gloss alignments and to verify Hands-On's training-data disjointness; this would settle whether the reported 0.7 BLEU-4 margin is attributable to SAGE's own contributions or to an uncontrolled pretrained segmentor.","tokens_in":12990,"tokens_out":7424,"duration_ms":89858,"concrete_test":"Obtain Hands-On's training-data manifest. If PHOENIX14T appears in any form (including PHOENIX14/PHOENIX14T train splits), rerun SAGE with (a) a segmentor trained only on disjoint sign data, and (b) a uniform tokenizer matching the same average token count (0.129); compare BLEU-4 on PHOENIX14T test. Independently, compute temporal boundary F1 between Hands-On segment predictions and PHOENIX14T gloss alignments on dev/test. If BLEU-4 drops by more than the reported 0.7 margin under the disjoint condition, or if boundary F1 is low, the SOTA/efficiency claim is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1.1 introduces the frozen Hands-On segmentor [25] as the sole basis for visual tokenization, and Section 4.3 credits the resulting 0.129 token ratio and +0.7 BLEU-4 (24.10 vs 23.40) to SAGE. The paper neither reports Hands-On's training corpus nor measures segment-boundary quality on PHOENIX14T; it only refers to supplementary materials that are absent from the arXiv v2 text. This matters because a sign-segmentation model is likely trained with gloss or boundary supervision on sign-language data. If that data includes PHOENIX14T (or its near-duplicates), then the tokenizer has already seen the benchmark's signing distribution, so the gloss-free framing and the performance/efficiency comparison are not a fair test of the proposed method. If the frozen segmentor produces inaccurate boundaries on PHOENIX14T, average-pooling within segments can discard linguistic content, yet no boundary F1 or per-segment token statistics are given. The central claim therefore rests on an uncontrolled variable; this is a missing-support gap, not a demonstrated failure, but it is load-bearing and checkable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SAGE, a two-stage gloss-free sign language translation (SLT) framework. In stage 1, a frozen segmentor (Hands-On) partitions continuous signing video into sign segments, each encoded via a ResNet-34 and a lightweight temporal encoder followed by average pooling to produce one visual token per segment. A token-to-token contrastive loss (CLCL) aligns these visual tokens with pseudo-glosses extracted from spoken sentences using POS tagging, with supervision applied at both the mBART token-embedding level and the hidden-state level. In stage 2, the pretrained visual encoder and mapper are attached to an mBART-large-50 encoder-decoder and fine-tuned for translation. On PHOENIX14T, the method reports a BLEU-4 of 24.10, a token reduction ratio of 0.129, and peak training memory of roughly 60 GB, claiming state-of-the-art gloss-free performance and improved scalability. Ablations evaluate loss choices, the weighting hyperparameter \\beta, and pretraining configurations on the development set.","tokens_in":13224,"tokens_out":4839,"duration_ms":53672,"significance":"If the claims hold, SAGE would be the first gloss-free SLT method to use sign segmentation as a visual tokenizer, achieving a meaningful reduction in input sequence length (about half of prior 0.25 ratios) while matching or exceeding current gloss-free BLEU scores. The idea of segment-aware tokenization with token-level contrastive alignment is a plausible and potentially scalable direction for SLT, and the paper is honest about some limitations, such as pseudo-gloss mismatches in the qualitative analysis. Strengths include the released code, clearly described architecture, internally consistent ablations, and explicit acknowledgment in Section 4.5 that spoken-language-derived pseudo-glosses do not always match signed content. However, the central empirical claims rest on an unverified frozen segmentor and on small performance differences without statistical support, so the significance is conditional on those points being resolved.","major_comments":[{"comment":"The frozen Hands-On segmentor [25] is load-bearing for both the claimed token compression (ratio 0.129 in Table 2) and the BLEU-4 improvement (24.10 vs. 23.40 in Table 1), but the paper does not report Hands-On's training corpus, nor any boundary-quality measure on PHOENIX14T. If Hands-On was trained on PHOENIX14T or its near-duplicates, the gloss-free framing and the comparison to prior work are not a fair test; if its boundaries are noisy, average pooling within segments can discard linguistic content. The manuscript defers these details to 'supplementary materials', but no supplementary file is present in the arXiv v2 submission. Please disclose the segmentor's training data, report segment-boundary quality (e.g., F1 against gloss-aligned boundaries or per-segment token statistics) on the PHOENIX14T test split, and, ideally, compare against a segmentor trained on a disjoint corpus.","section":"§3.1.1, §4.3, Table 1"},{"comment":"The central SOTA claim rests on a +0.7 BLEU-4 difference (24.10 vs. 23.40) over SignLLM, with no error bars, number of seeds, or statistical significance test. For sequence-to-sequence models on this benchmark, such a margin is within typical run-to-run variance; please provide mean and standard deviation over at least three seeds for SAGE and, if feasible, for the closest baselines, or a paired bootstrap test over test sentences.","section":"§4.3, Table 1"},{"comment":"The memory comparison reports '∼60 GB' and '∼3 × RTX 3090' as approximate values without specifying how peak VRAM was measured (e.g., per-component versus whole-training peak, batch size, sequence-length distribution). Because each baseline uses different hardware and batch sizes, the claimed 2.67× reduction is not on equal footing. Please report measured peak memory under matched conditions (same GPU, same batch size, comparable sequence lengths) or provide the full training configurations so the comparison can be reproduced.","section":"§4.3.2, Table 3"}],"minor_comments":[{"comment":"The notation t_i is used both for visual segment tokens in Section 3.1.1 and for subword tokens in Eq. (1); please use distinct symbols (e.g., s_i or u_i) to avoid ambiguity.","section":"§3.1.2, Eq. (1)"},{"comment":"The output dimension of the temporal encoder is written as R^{(n−4)×1024} because of the kernel size k=5, but the text does not explain that no padding is applied; please clarify this dimension change.","section":"§3.1.3"},{"comment":"The table captions do not state that the ablations are on the development set, although the text in Section 4.4 does; please make this explicit in the captions.","section":"§4.4, Table 4"},{"comment":"The paper refers to 'supplementary materials' in Sections 3.1.1, 4.2, and 4.5, but the arXiv v2 submission does not include a supplementary file; the final version must include the referenced material or clearly indicate where it is available.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim depends on the authors' own Hands-On segmentor, and the lack of transparency about its training data is the main risk to the validity of the SOTA comparison. The +0.7 BLEU-4 gap is small and unquantified, so the paper would benefit from a significance analysis. I recommend major revision focused on reproducibility and disclosure. The architecture and ablations are otherwise clear and the code release is a positive factor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: SAGE is the strongest gloss-free SLT result on PHOENIX14T I know of, and it earns that position with a genuinely sensible idea: use a frozen sign segmentor to turn video into a short sequence of segment tokens, then align those tokens to pseudo-glosses with token-level CLCL. The reported +0.7 BLEU-4 (24.10 vs 23.40) over SignLLM is modest, but the efficiency story — roughly half the token sequence and about 60 GB peak training memory — makes the combination worth taking seriously. This is not a paradigm shift; every component except the segment-aware tokenizer comes from prior work. But the segmentor-as-tokenizer application in gloss-free SLT is new, and the ablations (CLCL vs CLIP, dual supervision, transformer depth) are internally consistent and actually inform the design.\n\nThe soft spots are the ones the stress-test flags, and I think they are accurately proportioned. The load-bearing SOTA claim has no error bars or multi-seed runs, so a +0.7 BLEU-4 gap may not be significant. More important is the frozen Hands-On segmentor: same group, training corpus not disclosed, boundary quality on PHOENIX14T not measured, and the supplementary material mentioned in Section 3.1.1 is absent from the arXiv v2 text. If Hands-On was trained on PHOENIX14T or near-duplicates, the efficiency/accuracy comparison is not a fair test of the proposed tokenizer; if its boundaries are noisy, average pooling can discard sign content. The memory comparison mixes different GPUs, and the retrained GFSLT baseline lacks protocol detail. None of these are demonstrated failures — the method is coherent and the paper is honest about pseudo-gloss mismatches — but they are checkable missing-support gaps, and the central claim stays conditional until they are closed.\n\nWho should read this: people building practical gloss-free SLT systems, and anyone working on temporal compression for video-language models. I'd take it to reading group. I would not cite it in my own work yet, because the key numbers need the segmentor-data disclosure and artifact release to be trustworthy. But a serious editor should send this to reviewers; the idea is sound, the experiments are extensive, and the reproducibility conditions are clear. Reviewer asks: segmentor training data and overlap; boundary-quality metrics; multi-seed BLEU; and code/checkpoints. If those come back clean, the paper is a solid contribution.","headline":"SAGE is a solid, honestly-written efficiency contribution whose SOTA claim is conditional on disclosing the frozen segmentor's training data and adding error bars.","tokens_in":13812,"tokens_out":2838,"would_cite":false,"duration_ms":34206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Segmented video tokens enable state-of-the-art gloss-free sign language translation with roughly half the sequence length and 2.67x less memory than prior methods.","keywords":["sign language translation","gloss-free","visual tokenization","sign segmentation","contrastive learning","token-efficient","PHOENIX14T"],"falsifier":"Run SAGE with the Hands-On segmentor replaced by a random segmentation of the same average segment length on the PHOENIX14T dev set; if BLEU-4 does not drop substantially, the specific learned boundaries are not the source of the gain, contradicting the claim that segment-aware tokenization is the key mechanism.","tokens_in":12754,"feed_emoji":"🤟","tokens_out":7447,"duration_ms":76092,"temperature":0.7,"pith_summary":"This paper claims that sign language translation can be made both more accurate and much cheaper by first splitting continuous signing video into discrete segments, each corresponding to one sign, and translating each segment as a single visual token. On the PHOENIX14T benchmark, the resulting system, SAGE, reaches a BLEU-4 score of 24.10, beating the previous best gloss-free result of 23.40, while using roughly half the input sequence length and up to 2.67x less peak training memory than prior state-of-the-art. The authors argue that the efficiency gain is not a quality loss: even when previous methods are forced to similar sequence lengths, they fall behind. The paper also introduces a token-level contrastive alignment objective that is claimed to close the visual–language gap without any gloss annotations.","feed_headline":"Segment-aware tokens raise gloss-free sign translation BLEU-4 to 24.10","feed_subtitle":"Compressing video into per-sign tokens halves sequence length, cuts memory by 2.67x, and surpasses prior methods","key_machinery":"At the heart of the method is a segment-aware visual tokenizer: a frozen sign segmentation model (the Hands-On model) that predicts boundaries between signs from hand and body pose, followed by a hierarchical encoder that turns each sign segment into a single 1024-dimensional token. This tokenizer is what shrinks a video of hundreds of frames into a sequence of a few dozen tokens (reduction ratio ~0.129), making the Transformer's quadratic self-attention tractable. The alignment machinery is the token-level Cross-Lingual Contrastive Learning (CLCL) loss, applied at two levels of the language encoder—input embeddings and contextualized hidden states—to bind each visual token to a pseudo-gloss word. The final piece is the two-stage protocol: contrastive pretraining of the visual encoder and mapper, then fine-tuning with mBART for translation.","core_discovery":"SAGE introduces a two-stage gloss-free sign language translation framework. In the first stage, a frozen segmentation model splits continuous sign video into sign units, and a hierarchical visual encoder (ResNet-34 plus a 1D temporal convolution and average pooling) converts each unit into a single 1024-dimensional token; a transformer over these short token sequences captures cross-segment context. The authors align these visual tokens with pseudo-glosses—content words extracted from the spoken sentence via part-of-speech filtering—using a Cross-Lingual Contrastive Learning (CLCL) objective applied both to the mBART input embedding layer and to contextualized hidden states. In the second stage, the pretrained visual encoder and a visual-to-language mapper are attached to the mBART encoder–decoder and fine-tuned for translation. The central discovery is that segment-aware tokenization preserves the linguistic structure of signs while drastically shortening the input sequence, making state-of-the-art translation possible at a memory footprint of roughly 60 GB instead of 80–160 GB.","pith_inferences":["A learned, end-to-end boundary predictor optimized jointly with translation could outperform the frozen segmentor; the paper's own ablation discards the pretraining Transformer, suggesting the visual encoder itself may be simplified further.","Pseudo-gloss extraction via POS filtering may be replaced by a learnable text tokenizer or by mBART's own subword units, which could better handle multi-word signs and reduce the mismatches shown in the similarity matrix.","The 0.129 reduction ratio is not a hard floor: co-articulated signs with redundant frames could be merged, potentially halving token count again without loss.","If SAGE transfers to continuous sign language recognition, segment tokens could replace frame-wise CTC alignment, since gloss boundaries are typically weak supervision in that task."],"forward_implications":["A single ~60 GB GPU budget suffices for state-of-the-art gloss-free training, roughly 2.67x less memory than Sign2GPT and 1.33x less than GFSLT.","At reduction ratio 0.129, the token sequence is about half the length used by comparable methods, cutting the Transformer's quadratic attention cost proportionally.","Token-level CLCL alignment improves BLEU-1 through BLEU-3 more than global CLIP-style alignment, indicating finer-grained visual-to-word correspondence.","Under matched half-length inputs, SAGE's BLEU-4 of 24.10 exceeds GFSLT's 19.25 and FLa-LLM's 20.02, showing that semantic tokenization retains quality where uniform downsampling degrades it."],"supporting_citations":[{"why":"Supplies the frozen sign segmentation model that turns video into sign segments, the core of the tokenizer.","marker":"[25]"},{"why":"Provides the token-level cross-lingual contrastive loss (CLCL) used to align visual tokens with pseudo-glosses.","marker":"[9]"},{"why":"Is the main gloss-free baseline and prior state-of-the-art; its pretraining pipeline and results are reproduced for comparison.","marker":"[45]"},{"why":"Introduces the pseudo-gloss generation via POS tagging and the Sign2GPT baseline that SAGE surpasses.","marker":"[37]"},{"why":"Is the previous best gloss-free method (BLEU-4 23.40) that SAGE beats by +0.7.","marker":"[12]"},{"why":"Reports the T/8 downsampling result used to show SAGE's advantage under matched sequence lengths.","marker":"[8]"},{"why":"Defines the PHOENIX14T benchmark and the neural sign language translation task.","marker":"[3]"},{"why":"Provides the mBART-large-50 language encoder used for pseudo-gloss embeddings and the final translation decoder.","marker":"[35]"}],"fun_headline_variants":["Sign translation with 50% fewer tokens beats prior methods","Gloss-free SLT: 2.67x memory cut, BLEU-4 24.10","Segment-aware tokens slash sequence length, improve translation","SAGE: token-efficient sign translation without glosses","Halving input length lifts sign translation BLEU-4"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen Hands-On segmentation model, without any fine-tuning, produces reliable, semantically meaningful sign boundaries on PHOENIX14T; if those boundaries are noisy or the model's training data overlaps the test split, the token compression loses linguistic content and the efficiency–accuracy trade-off collapses.","fun_headline_variants_meta":{"raw":{"variants":["Sign translation with 50% fewer tokens beats prior methods","Gloss-free SLT: 2.67x memory cut, BLEU-4 24.10","Segment-aware tokens slash sequence length, improve translation","SAGE: token-efficient sign translation without glosses","Halving input length lifts sign translation BLEU-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000758,"raw_usage":{"total_tokens":3373,"prompt_tokens":958,"completion_tokens":2415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2324}},"tokens_in":574,"tokens_out":2415,"duration_ms":17966,"temperature":1.0,"reasoning_tokens":2324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:00:26.868406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SAGE with the Hands-On segmentor replaced by a random segmentation of the same average segment length on the PHOENIX14T dev set; if BLEU-4 does not drop substantially, the specific learned boundaries are not the source of the gain, contradicting the claim that segment-aware tokenization is the key mechanism.","supporting_citations":[{"cited_title":"Hands-on: Segmenting individual signs from continuous sequences","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen sign segmentation model that turns video into sign segments, the core of the tokenizer."},{"cited_title":"Cico: Domain-aware sign language re- trieval via cross-lingual contrastive learning","cited_arxiv_id":null,"evidence_quote":"Provides the token-level cross-lingual contrastive loss (CLCL) used to align visual tokens with pseudo-glosses."},{"cited_title":"Gloss-free sign language translation: Improving from visual- language pretraining","cited_arxiv_id":null,"evidence_quote":"Is the main gloss-free baseline and prior state-of-the-art; its pretraining pipeline and results are reproduced for comparison."},{"cited_title":"Sign2gpt: Leveraging large language models for gloss-free sign language translation","cited_arxiv_id":null,"evidence_quote":"Introduces the pseudo-gloss generation via POS tagging and the Sign2GPT baseline that SAGE surpasses."},{"cited_title":"Llms are good sign language translators","cited_arxiv_id":null,"evidence_quote":"Is the previous best gloss-free method (BLEU-4 23.40) that SAGE beats by +0.7."},{"cited_title":"Factorized learn- ing assisted with large language model for gloss-free sign language translation","cited_arxiv_id":null,"evidence_quote":"Reports the T/8 downsampling result used to show SAGE's advantage under matched sequence lengths."},{"cited_title":"Neural sign language trans- lation","cited_arxiv_id":null,"evidence_quote":"Defines the PHOENIX14T benchmark and the neural sign language translation task."},{"cited_title":"Mul- tilingual translation with extensible multilingual pretraining and finetuning","cited_arxiv_id":null,"evidence_quote":"Provides the mBART-large-50 language encoder used for pseudo-gloss embeddings and the final translation decoder."}],"review_version":1}