{"id":"e4059004-ddc3-41c8-9cc6-1d6217207e3a","arxiv_id":"2608.11332","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Weak transcript-derived pseudo-gloss pretraining on Turkish broadcast data raises cross-dataset sign spotting top-5 mean IoU from 0.235 to 0.465 and improves translation BLEU-4 from 9.60 to 11.04.","lead":"This paper tests whether Turkish broadcast news transcripts, converted into rough word labels without expert annotation, can teach a video encoder to spot dictionary signs inside continuous signing. The work offers a low-cost path to sign-language tools for languages with little annotated data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gain cannot be attributed to pseudo-glosses because the only no-pretraining baseline lacks the temporal encoder, and no random-label control is run.","rationale":"The reader's weakest assumption focuses on the frozen LLM lexical-rule mapping not being consistency-audited. That is a reproducibility concern, but the paper explicitly tolerates noisy pseudo-glosses, and the central claim would survive a certain amount of mapping noise as long as the noise is not systematic in a way that corrupts the evaluation. A more load-bearing threat is that the main quantitative evidence for the central claim is not yet isolated from architectural factors. The only nontrained baseline, raw spatial DINOv2 features, lacks the temporal encoder entirely, so the jump from 0.235 to 0.465 cannot be cleanly assigned to pseudo-gloss supervision. The absence of a random-label or untrained-temporal-encoder control leaves open the possibility that the gain comes from in-domain temporal feature adaptation rather than from text-derived lexical structure. This is a concrete, testable gap rather than a demonstrated error: the LLM-versus-morphology shared-subset improvement gives real partial evidence that label content matters, and the frequency analysis helps rule out simple label memorization. For those reasons I do not think the paper should be rejected or marked unverdictable; it should remain conditional, with the required condition being a random-label control rather than an LLM consistency audit. I therefore keep the reader's CONDITIONAL verdict, and I partially agree with the reader's stated weakest assumption: it identifies a valid limitation, but not the single most load-bearing one for the paper's central attribution claim.","tokens_in":12631,"tokens_out":5887,"duration_ms":60368,"concrete_test":"Train the stage-1 encoder on TSL-News with randomly shuffled pseudo-gloss labels, preserving the 6539-class vocabulary and per-video bag sizes but permuting labels across videos, and evaluate the resulting encoder with the Table 3 protocol. As a second condition, evaluate the same stage-1 architecture with random initialization and no training. If the random-label top-5 mean IoU is within about 0.02 of 0.465, the reported gain is not attributable to lexical content; if it falls to near the spatial baseline, the pseudo-gloss text is doing real work. This check uses one extra training run and the existing evaluation pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 3, the spatial-feature baseline uses raw DINOv2 frame features without the trainable MetaFormer temporal encoder, temporal downsampling, or LoRA adaptation that are present in every pretrained encoder (Sec. 4.3, Appendix A.1). The reported improvement from 0.235 to 0.465 top-5 mean IoU therefore conflates pseudo-gloss supervision with architectural and temporal-processing changes introduced by stage-1 training. The paper never trains the same encoder with shuffled or random pseudo-gloss labels, nor evaluates an untrained version of the temporal encoder, so it cannot rule out that any in-domain TSL-News training with temporal modeling improves NCC-based matching. The shared-subset comparison between the morphology-lemma and LLM encoders (0.368 vs. 0.473) does show that label quality matters when the architecture is fixed, which is partial evidence in favor of the central claim. However, Table 6 substantially weakens the lexical interpretation: the pseudo-gloss classifier itself achieves only 0.280 top-5 IoU, while target-known NCC reaches 0.465. The strong target-known NCC may therefore reflect temporally smoothed visual similarity rather than a lexical-temporal representation aligned with sign meaning. A random-label control is needed to separate these explanations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether weak transcript-based supervision from broadcast news can pretrain a reusable sign-language video encoder under Turkish's agglutinative morphology. The authors introduce TSL-News, a Turkish broadcast corpus with sentence-level transcripts, and TSL-SB, a dictionary-based spotting benchmark built from the TSLD corpus. They compare two pseudo-gloss construction strategies from text: rule-based morphological lemmatization and a constrained LLM-assisted lexical normalization over a fixed vocabulary. The stage-1 encoder uses DINOv2 frame features with LoRA adaptation, a MetaFormer temporal encoder, and a fastText-initialized prototype head trained with a sentence-level BCE loss. Evaluation uses normalized cross-correlation (NCC) between isolated dictionary templates and continuous example video representations, reporting top-k temporal IoU. The LLM lexical-rule encoder is reported to improve top-5 mean IoU from 0.235 (raw spatial features) to 0.465, with 56.2% of examples reaching IoU > 0.50. A shared-subset comparison between the two pseudo-gloss strategies shows gains of about 0.10 IoU for the LLM strategy. The paper also reports a downstream translation check where pretraining improves BLEU-4 from 9.60 to 11.04. The central claim is that noisy sentence-level pseudo-glosses can shape a visual encoder into a lexical-temporal representation that transfers cross-dataset for sign spotting.","tokens_in":12867,"tokens_out":5988,"duration_ms":63171,"significance":"If the claims hold, the paper offers a practical weak-supervision route for low-resource sign languages and introduces two potentially useful resources (TSL-News and TSL-SB). The diagnostic protocol is thoughtful: the shared-subset comparison in Sec. 5.1 controls for vocabulary-coverage differences, the frequency analysis in Sec. 5.1 addresses memorization, and the multi-angle evaluation in Sec. 5.3 attempts to separate representation quality from classifier bias. The main unresolved threats are (1) the no-pretraining baseline lacks the temporal encoder and other architectural components, so the headline gain is not attributable to pseudo-gloss supervision; (2) the absence of a random-label or untrained same-architecture control; (3) single-seed reporting without variance; and (4) the benchmark only contains signs present in pretraining, limiting the cross-dataset claim. These are fixable with additional experiments, and the shared-subset comparison already provides partial evidence that label quality matters.","major_comments":[{"comment":"The spatial-feature baseline used in Table 3 is not a controlled ablation of pseudo-gloss supervision: it lacks the trainable MetaFormer temporal encoder, temporal downsampling, and LoRA adaptation present in every pretrained encoder (Sec. 4.3, Appendix A.1). The reported gain from 0.235 to 0.465 top-5 IoU therefore conflates supervision with architectural and temporal-processing changes. To attribute the gain to pseudo-glosses, provide a same-architecture no-pretraining baseline, e.g., the MetaFormer encoder with random or frozen DINOv2 features evaluated without stage-1 training, or a control trained with shuffled/random pseudo-gloss labels. The shared-subset comparison in Sec. 5.1 controls coverage but not this architecture confound.","section":"Sec. 4.3, Table 3"},{"comment":"The pseudo-gloss classifier diagnostic achieves only 0.280 top-5 IoU versus 0.465 for target-known NCC on the same LLM lexical-rule benchmark. This large gap suggests that the dominant matching signal in NCC may be temporally smoothed visual similarity rather than lexical alignment. Please discuss this discrepancy explicitly and, if possible, report target-known NCC for the same-architecture no-training baseline. Such a comparison would help identify whether the NCC gain arises from learned lexical-temporal structure or from temporal smoothing alone.","section":"Sec. 5.3, Table 6"},{"comment":"All experiments use a single seed (seed 1). No variance or statistical significance measures are reported across Tables 3-7. Because the central claims are comparative improvements (LLM vs. morphology vs. no pretraining), single-seed results are insufficient to establish reliability. Please run at least three seeds with the same protocol and report means and standard deviations, or provide a clear justification for why the reported differences are beyond stochastic variation.","section":"Appendix A.1, Tables A.1 and A.2"},{"comment":"The benchmark by construction evaluates only signs that appear at least once in the TSL-News pretraining text, as the paper acknowledges in the Fig. 3 caveats. This limits the 'cross-dataset' claim to within-vocabulary transfer and leaves open what happens for unseen signs. Please state this limitation prominently in the abstract and conclusion, and consider adding a small held-out vocabulary test (e.g., signs present in TSL-SB but excluded from TSL-News text) to probe generalization to unseen lexical items.","section":"Sec. 5.1, Fig. 3"}],"minor_comments":[{"comment":"The statement 'raises top-5 mean IoU from 0.235 to 0.465' should clarify that the 0.235 baseline uses raw spatial features without the stage-1 architecture; as written, the abstract implies a controlled comparison.","section":"Abstract, Sec. 1"},{"comment":"The admission that 'no independent repeat-call consistency audit of the original local LLM server was logged' is important; please add a sentence in the main text or supplement acknowledging this as a limitation of the LLM-based normalization, since the frozen mapping may still contain inconsistencies.","section":"Appendix A.3"},{"comment":"The Spearman correlation of -0.069 is reported without a p-value or confidence interval; please report the p-value or use a permutation test to support the claim that frequency is not associated with localization quality.","section":"Fig. 3"},{"comment":"The row for the spatial-feature baseline uses the same example set as the LLM lexical-rule encoder (N=1817, Skip=25), but it does not share the architecture; a note in the table caption would prevent misinterpretation.","section":"Table 3"},{"comment":"The definition of p_g in Eq. (2) is clear in principle, but it would help to explicitly state that softmax_g is over the vocabulary dimension and softmax_t' over the time dimension, and that both use the same input s_{t',g} scaled by their respective temperatures.","section":"Sec. 3.1, Eq. (2)"},{"comment":"The symbol N denotes example groups in Table 3 but segment occurrences in the all-vocabulary row of Table 6; please make the definition explicit in each table caption.","section":"Tables 3 and 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution to an underexplored problem, and the shared-subset analysis is a step in the right direction. However, the main headline is currently based on an uncontrolled comparison, and the single-seed reporting is a common but important weakness. The required fixes—a same-architecture no-training or random-label baseline, multi-seed runs, and clearer framing of the vocabulary limitation—are feasible within the manuscript's scope. I also note that the frozen LLM mapping is a key artifact; releasing it alongside the corpora would substantially aid reproducibility, since the original local server is not available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. First, it's a genuinely useful contribution for low-resource sign language: a new Turkish broadcast corpus (TSL-News), a new dictionary-based spotting benchmark (TSL-SB), and a constrained LLM pseudo-gloss normalization scheme that beats rule-based lemmatization on a controlled shared subset. Second, the headline improvement from 0.235 to 0.465 top-5 IoU is not cleanly attributable to pseudo-glosses, because the only no-pretraining baseline uses raw spatial features without the temporal encoder or LoRA adaptation that every trained model has. There is no random-label control, so you can't rule out that any in-domain training with temporal modeling would produce a large part of that gain.\n\nThe paper's strongest evidence is the shared-subset comparison in Section 5.1: on the same 1137 examples, the LLM-based encoder gets 0.473 top-5 IoU versus 0.368 for the morphology-lemma encoder, holding architecture fixed. That shows label quality matters. The frequency analysis is a decent check against memorization, and the paper is admirably honest about its limitations: the benchmark only tests signs that appear in pretraining, the all-vocabulary diagnostic is restricted to annotated intervals, and the LLM mapping's consistency was never audited.\n\nThe main soft spots are practical. Single seed, no error bars. The target-known NCC is much stronger than the direct pseudo-gloss classifier (0.465 vs 0.280), which raises the question of whether the encoder learned lexical-temporal structure or just temporally smoothed visual similarity. A random-label control with the same architecture would answer that. The datasets are not yet released, which limits immediate reuse.\n\nThe stress-test note is correct about the confound, but it's a fixable omission, not a load-bearing flaw. The paper deserves a serious referee. I'd ask for a random-label baseline, multi-seed variance, and a release plan for code and data. If the datasets come out, I'd likely cite them. I'd bring this to a reading group focused on sign language or weak supervision, because the methodological issues are instructive.","headline":"Weak supervision can help sign spotting in Turkish, but the headline gain is confounded by a missing architecture control; the shared-subset comparison is the real result.","tokens_in":13440,"tokens_out":4075,"would_cite":true,"duration_ms":36101,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Weak transcript-derived pseudo-glosses can pretrain a reusable Turkish Sign Language encoder that transfers to cross-dataset sign spotting, nearly doubling top-5 temporal localization IoU over raw spatial features.","keywords":["Turkish Sign Language","sign spotting","pseudo-glosses","weak supervision","representation learning","temporal localization","broadcast news","agglutinative morphology"],"falsifier":"Run the same word-to-pseudo-gloss mapping repeatedly on a held-out set of Turkish broadcast transcripts and measure the agreement rate; or evaluate the pretrained encoder on TSLD signs whose Turkish lexical forms never appear in TSL-News transcripts—if top-5 IoU collapses for those unseen signs, the representation is overfit to frequent transcript words rather than generalizing via sign structure.","tokens_in":12445,"feed_emoji":"👋","tokens_out":5509,"duration_ms":58767,"temperature":0.7,"pith_summary":"The paper asks whether noisy sentence-level transcripts, not manual glosses, can pretrain a reusable sign-language video encoder for Turkish Sign Language, a morphologically rich and resource-limited setting. It claims the answer is yes: an encoder shaped by weak pseudo-gloss labels derived from broadcast news transfers across datasets to localize isolated dictionary signs inside continuous sentence videos. The central comparison pits rule-based morphological lemmatization against a constrained LLM lexical normalizer, with the LLM version raising top-5 mean IoU from 0.235 to 0.465 and lifting the share of examples reaching an IoU of at least 0.50 from 23.9% to 56.2%. Because only the representation changes, not the spotting architecture, the result argues that text-derived pseudo-glosses carry genuine lexical-temporal signal.","feed_headline":"Transcript-filtering trick lifts sign-spotting IoU to 0.465","feed_subtitle":"Noisy sentence-level Turkish transcripts train a reusable encoder, more than doubling top-5 temporal localization over raw features.","key_machinery":"The load-bearing mechanism is pseudo-gloss pretraining with a two-temperature temporal-class scoring rule combined with normalized cross-correlation (NCC) matching. The sentence-level score for each pseudo-gloss aggregates per-frame similarities by taking a softmax over both pseudo-gloss classes and over time steps (Eq. 2); because no temporal target is supplied, any temporal ordering in the hidden states must emerge from the video dynamics, the encoder's inductive bias, and the weak label pressure. The second load-bearing component is the constrained LLM lexical-rule mapping, which selects candidate pseudo-glosses from a fixed inventory, canonicalizes inflectional and spelling variants while preserving derivational distinctions, and thereby raises overlap with the dictionary vocabulary from 52.89% to 69.76%.","core_discovery":"On the paper's own terms, the discovery is that even noisy, sentence-level pseudo-glosses—bags of Turkish words extracted from transcripts without temporal boundaries—can shape a frozen-backbone video encoder into a lexical-temporal representation strong enough to support cross-dataset sign spotting. The encoder is pretrained on the new TSL-News broadcast corpus with a binary cross-entropy loss over pseudo-gloss presence, using a prototype head initialized from fastText subword embeddings, and then evaluated by normalized cross-correlation between isolated dictionary sign templates and continuous sentence hidden states on the new TSL-SB benchmark. The constrained LLM-assisted normalization, which maps surface forms onto a fixed vocabulary of 6,539 pseudo-glosses, outperforms both raw spatial features and the rule-based morphological lemmatization vocabulary of 4,802 on the full benchmark and on the 1,137 examples shared by both vocabularies. The frequency analysis (Spearman rho = -0.069) supports the claim that the gains are not mainly memorization of frequent pseudo-gloss labels.","pith_inferences":["A natural test that the paper does not run: feed the same encoder to a proper temporal-proposal generator and check whether the representation lifts open-vocabulary spotting, not just interval-constrained matching.","The closed-vocabulary LLM normalization could be replaced by an unsupervised subword segmenter, removing the need for a fixed candidate inventory and easing transfer to new broadcast domains without manual review.","The rule-based analyzer and the LLM mapper make different errors, so a hybrid that consults the analyzer only when the LLM rejects all candidates might reduce supervision noise further.","If the representation is truly lexical-temporal, it should be usable for sign-language retrieval and dictionary mining beyond spotting; the paper's future-work list points there but does not test retrieval."],"forward_implications":["Cross-dataset sign spotting becomes feasible for languages that have broadcast data but no expert gloss annotations: a single pretrained encoder can serve dictionary-query spotting on a held-out corpus.","Constrained LLM normalization over a fixed vocabulary is a viable alternative to morphological analyzers for agglutinative languages, buying both vocabulary coverage and localization accuracy.","Downstream translation starts from a better visual representation: reusing the pretrained encoder improves BLEU-4 from 9.60 to 11.04 and ROUGE from 23.48 to 27.43.","The weak supervision signal is chiefly lexical-temporal rather than frequency-based, since localization quality is nearly flat across train-sentence frequency quartiles.","The representation rather than the classifier carries the spotting signal: all-vocabulary local NCC outperforms direct pseudo-gloss score localization (top-5 IoU 0.516 versus 0.280)."],"supporting_citations":[{"why":"Supplies the pseudo-gloss pretraining architecture that the paper adapts and repurposes for representation learning.","marker":"[20]"},{"why":"Provides the frozen DINOv2 frame backbone used for spatial features.","marker":"[14]"},{"why":"Defines the normalized cross-correlation metric used for temporal localization.","marker":"[8]"},{"why":"Establishes the dictionary-query spotting view with weak subtitle supervision.","marker":"[11]"},{"why":"Provides the finite-state Turkish morphology analysis underlying the rule-based lemma strategy.","marker":"[12]"},{"why":"Supplies morphological disambiguation techniques that the rule-based strategy draws on.","marker":"[13]"},{"why":"Demonstrates large-vocabulary continuous sign recognition from spoken-language supervision, the strongest prior baseline for weak-supervision spotting.","marker":"[16]"},{"why":"The TSLD dictionary corpus anchors the benchmark and supplies the isolated sign templates.","marker":"[10]"}],"fun_headline_variants":["Transcript-derived pseudo-glosses lift sign-spotting IoU to 0.465","Weak transcript supervision nearly doubles sign-spotting IoU","Noisy broadcast transcripts train reusable sign encoder","LLM-assisted normalization boosts cross-dataset sign spotting","Sign-spotting IoU more than doubles with transcript pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The frozen, manually reviewed LLM word-to-pseudo-gloss mapping is correct and consistent; the paper states that no independent repeat-call consistency audit of the original local LLM server was logged, so systematic misassignment of Turkish surface words could corrupt the training targets and make the reported transfer gains reflect text-matching artifacts rather than sign structure.","fun_headline_variants_meta":{"raw":{"variants":["Transcript-derived pseudo-glosses lift sign-spotting IoU to 0.465","Weak transcript supervision nearly doubles sign-spotting IoU","Noisy broadcast transcripts train reusable sign encoder","LLM-assisted normalization boosts cross-dataset sign spotting","Sign-spotting IoU more than doubles with transcript pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1926,"prompt_tokens":1091,"completion_tokens":835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":749}},"tokens_in":707,"tokens_out":835,"duration_ms":25305,"temperature":1.0,"reasoning_tokens":749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:56.392310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same word-to-pseudo-gloss mapping repeatedly on a held-out set of Turkish broadcast transcripts and measure the agreement rate; or evaluate the pretrained encoder on TSLD signs whose Turkish lexical forms never appear in TSL-News transcripts—if top-5 IoU collapses for those unseen signs, the representation is overfit to frequent transcript words rather than generalizing via sign structure.","supporting_citations":[{"cited_title":"In: Vision interface","cited_arxiv_id":null,"evidence_quote":"Defines the normalized cross-correlation metric used for temporal localization."},{"cited_title":"In: Proceedings of the Asian conference on computer vision (2020)","cited_arxiv_id":null,"evidence_quote":"Establishes the dictionary-query spotting view with weak subtitle supervision."},{"cited_title":"In: Fourth Conference on Applied Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Provides the finite-state Turkish morphology analysis underlying the rule-based lemma strategy."},{"cited_title":"In: Fourth Conference on Applied Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Supplies morphological disambiguation techniques that the rule-based strategy draws on."},{"cited_title":"A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision","cited_arxiv_id":"2405.10266","evidence_quote":"Demonstrates large-vocabulary continuous sign recognition from spoken-language supervision, the strongest prior baseline for weak-supervision spotting."},{"cited_title":"(eds.): Güncel Türk İşaret Dili Sözlüğü","cited_arxiv_id":null,"evidence_quote":"The TSLD dictionary corpus anchors the benchmark and supplies the isolated sign templates."}],"review_version":1}