{"id":"1ead1af6-1ba7-4e0e-8967-ccfd119b0537","arxiv_id":"2507.21104","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"iLSU-T is a 187-hour Uruguayan Sign Language video dataset with Spanish text, 18 interpreters, and first baseline translation results.","lead":"Researchers assembled 187 hours of Uruguayan Sign Language video from public TV and Parliament, paired with Spanish text and audio, and ran three translation models as a first benchmark. It is the first translation-oriented dataset for this sign language, giving researchers a local resource for sign-to-Spanish translation work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automatic clip-text alignment uses unvalidated random delays (Section IV-B1); if the true interpreter delay varies by source or signer, every training/test clip and all reported BLEU/ROUGE baselines are misaligned, and the paper's 20+ hours of manual alignments were not used to test this.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I find: the random-delay clip-text alignment is unvalidated. My read agrees with that assessment and with the conditional verdict. I credit the paper for releasing code, collecting over 20 hours of manual alignments, and transparently discussing duplicate-phrase effects and ASR errors. However, the alignment step is the linchpin of the dataset and the benchmark: if the delay distribution is wrong, the text labels are not ground truth and the reported translation scores are not trustworthy. Because the manual alignments already exist, the proposed check is cheap, decisive, and directly tests the paper's own acknowledged limitation. This does not justify rejection, since the dataset resource and pipeline remain valuable, but it does justify keeping the verdict conditional until the alignment is validated.","tokens_in":15302,"tokens_out":3437,"duration_ms":40699,"concrete_test":"Use the subset of episodes with expert manual alignment (Section III-D) to compute, for each aligned phrase, the true pre-delay (manual video start minus text start) and post-delay (video end minus text end). Compare these empirical distributions to U(0.4,1.2) and U(2.1,2.9), stratified by source and signer. If the empirical means or 90% intervals differ by more than about 0.3 s, or if a source-by-signer interaction appears, regenerate Source-3 clips using manual boundaries and rerun the STLCU I3D-ASL2k baseline; if BLEU-4 on the test set shifts by more than 0.5 points, the random-delay clipping is not a safe basis for the reported benchmark. If the metrics are materially unchanged, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that iLSU-T is a usable, large-scale LSU translation dataset with a meaningful first baseline. That requires each video clip to contain the signed rendering of its paired Spanish sentence. The pipeline in Section IV-B1 generates clips by sampling t1~U(0.4,1.2) and t2~U(2.1,2.9) seconds, chosen by visual inspection from a random sample of episodes, and applies this same model to all sources and signers. No validation against the expert manual alignments (Section III-D, 'over 20 hours') is reported, even though those alignments could supply ground truth for the delay distribution. The paper itself concedes in Section VI that video-clip conformation is based on 'empirically adjusted random delays' and is 'a first approximation,' and it calls automatic text-video alignment an open problem. Because the three sources have different interpreting styles and signer sets, a single delay distribution risks systematically corrupting clip boundaries, making the paired text partially or wholly wrong for a large fraction of the 86,550 clips. Since all reported BLEU/ROUGE numbers are computed on these clips, the benchmark could substantially misrepresent method performance. This is not a disagreement with consensus; it is an internal validity issue that the paper acknowledges but does not test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces iLSU-T, a publicly released multimodal dataset for Uruguayan Sign Language (LSU) translation, built from 571 episodes of interpreted TV and parliamentary video totaling 187.4 hours, with 18 signers, three sources, WhisperX-derived Spanish text, linguistic topic and genre labels, and over 20 hours of expert manual text-video alignment. It describes a preprocessing pipeline (RoI annotation, signer identification, automatic captioning, manual phrase alignment, contextual labeling) and evaluates three gloss-free sign language translation methods (SLT, STLCU, GASLT) with two I3D feature extractors on four data configurations, reporting BLEU and ROUGE-L baselines. The authors frame iLSU-T as the first large-scale LSU translation resource and the reported numbers as the first LSU benchmark.","tokens_in":15580,"tokens_out":5237,"duration_ms":60253,"significance":"If the clip-text pairing is trustworthy, the dataset fills a real gap: no large-scale public LSU translation resource exists, and the three-source structure with different signers and backgrounds is valuable for studying domain shift. Strengths include the public release with configuration files, the expert manual alignment subset, the linguistic context labeling, and the candid discussion of known limitations of ASR-derived text, fingerspelling, and coreference. The dataset's scientific value, however, rests on the validity of the automatic text-video association used to create the 86,550 training clips; the paper's own caveats in Section VI make that the central risk. The baseline numbers are conditional on the clipping procedure, so they should be treated as provisional until the alignment is validated. The contribution is significant for the low-resource sign language translation community if that validation is supplied.","major_comments":[{"comment":"The entire benchmark depends on the random-delay clipping procedure, but the procedure is never validated. Section IV-B1 sets t1 ~ U(0.4,1.2) and t2 ~ U(2.1,2.9) seconds for all sources and signers based on visual inspection of a random sample of episodes, and the paper calls this a first approximation and labels automatic text-video alignment an open problem (Section VI). Because Sources 1-3 involve different broadcast formats and interpreting styles, a single delay distribution can systematically produce clips whose paired Spanish text does not correspond to the signed content, which would invalidate every BLEU/ROUGE value in Tables III-VI. The authors already have the tool needed to test this: Section III-D reports over 20 hours of expert manual alignments, but no comparison between automatic and manual clip boundaries is reported. I ask for a quantitative validation on the manually aligned episodes (e.g., distribution of delay offsets per source and signer, agreement between automatic and manual boundaries, and re-estimated delay parameters), plus an analysis of how much the final metrics change when using the manual alignments as test references. Without this, the dataset's core clip-text pairs remain unverified.","section":"IV-B1 and VI"},{"comment":"The performance gap on Source-3 is partly explained by duplicate text phrases between splits, but the problem is not confined to Source-3. Section VI states that Source-3 includes duplicate text phrases between train, validation, and test, and Figure 5 shows that duplicate phrases score dramatically higher (e.g., BLEU-4 27.89 vs 3.09). However, Section IV-B1's claim that the text content of clips is a priori independent is insufficient because identical or near-identical Spanish sentences recur within and across episodes (parliamentary protocol phrases, news items), and random clip-level splitting can place them in different splits. The whole-dataset and Source-1/Source-2 results in Tables III-V are therefore also exposed to this leakage unless explicit deduplication was performed. Please report the fraction of test (and validation) clips whose reference sentence or a near-duplicate appears in the training set for all four configurations, and provide results on a deduplicated split. This is required to interpret the Source-3 advantage and the general baselines.","section":"VI and Fig. 5"},{"comment":"The reference text is produced by WhisperX automatic speech recognition without a reported word error rate, and the paper itself gives an example of a mistranscribed sentence (a ese cuenta de otra solicitud y licencia llegada a la mesa, Table VII discussion) and notes punctuation errors that affect clipping. Because BLEU and ROUGE-L compare against these noisy references, the absolute scores in Tables III-VI are not trustworthy as translation quality estimates, and punctuation errors also alter t1/t2-based clip boundaries. I recommend quantifying ASR quality (e.g., WER/CER on a human-transcribed subset, including the over-20-hours of manual alignments) and reporting at least one evaluation using manually corrected transcripts on the test set. This would also clarify how much of the reported gap is due to the translation models versus the reference quality.","section":"III-B3 and VI"},{"comment":"Tables III-VI report single-run scores with random splits and random delays, but no seeds, confidence intervals, or variance estimates are given. Because the data splits and the clipping delays are stochastic, the reported differences between methods and feature extractors (e.g., Table III STLCU vs GASLT) may be within run-to-run noise. Please provide seeds or multiple runs with mean and standard deviation, at least for the main configurations, so the baseline comparisons are reproducible and interpretable.","section":"IV-B2 and Tables III-VI"}],"minor_comments":[{"comment":"The dataset name appears as iLSU T in the abstract and iLSU-T elsewhere; please standardize. Also, Table I lists 201.5 hours while Section III-C reports 187.4 hours and Section IV-B1 reports 201.52 hours as the total duration of all video clips; clarify that the latter is the overlapping sum of clips while the episode-level curated content is 187.4 hours.","section":"Abstract and Table I"},{"comment":"The method name is misspelled as SCULT in the table header; it should be STLCU.","section":"Table VII"},{"comment":"The statement that substituting word encoding for BPE performs better is made without supporting numbers; please add a short comparison or remove the claim.","section":"IV-A3"},{"comment":"The KNN classifier uses 50 samples per signer and a median filter with a 30-second minimum signer duration; please state explicitly whether these values were tuned and list them in the reproducibility notes, like the delay parameters in IV-B1.","section":"III-B2"},{"comment":"The ROUGE-L formula uses beta=1.2 in a non-standard F-measure form; since all methods use the same official implementation this is not a correctness issue, but please define ROUGE-L consistently with the cited reference [26].","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent and the dataset is potentially valuable, but the benchmark claims need the alignment validation, leakage analysis, ASR-reference quality assessment, and run-variance reporting described in the major comments. These are fixable with additional experiments that the existing manual alignments already enable, so I do not see grounds for rejection; the contribution should be accepted only after the validity of the clip-text pairs is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the resource is real: iLSU-T is the first large-scale translation-oriented dataset for Uruguayan Sign Language, with 187 hours, 18 interpreters, three sources, and a documented pipeline. That alone is a contribution the field can build on. The authors also ship code, describe the curation stages in enough detail to reproduce, and provide 20+ hours of expert manual phrase alignments, which is more ground truth than many sign language datasets offer. The baselines are standard state-of-the-art methods, run honestly with SacreBLEU, and the qualitative examples are informative, especially the BERTScore discussion.\n\nThe weak point is the automatic video-clip/text alignment. Section IV-B1 samples pre/post delays from uniform distributions chosen by visual inspection from a sample of episodes, and the same distribution is applied to all sources and signers. The paper itself calls this \"a first approximation\" and says automatic alignment is an open problem, so the authors are not hiding it. But they had over 20 hours of manual alignments and never used them to validate or calibrate the delay distribution. That is a missed opportunity, and it matters: if the delay varies by source or signer, a large fraction of the 86,550 clips could be paired with the wrong Spanish text, and all reported BLEU/ROUGE numbers inherit that error. The benchmark should be read as provisional until this is checked.\n\nThe other issues are real but less severe. Duplicate phrase leakage across splits inflates the Source-3 numbers; the authors acknowledge this and even quantify it in Figure 5, which is good practice, though it means the Source-3 absolute scores shouldn't be compared with other datasets. There are no seeds or error bars, so small differences between methods are not meaningful. ASR transcription errors are acknowledged and visible in the examples; that is a known limitation of subtitle-derived data, not a hidden flaw.\n\nOverall, the central contribution, the dataset, holds up. The benchmark numbers are the part that needs a health warning. This is a resource paper, not a methods paper, and it should be judged that way. Who it's for: anyone working on low-resource sign language translation, especially for Latin American sign languages, will want to use or cite iLSU-T. The authors have been commendably open about limitations, and the reproducibility pipeline is real. I'd send this to peer review. It needs revision, but it's the right kind of paper: a first solid resource for a language that had none.","headline":"A genuinely useful first dataset for a low-resource sign language, with benchmark numbers that are provisional until the clip-alignment heuristic is validated.","tokens_in":16109,"tokens_out":1671,"would_cite":true,"duration_ms":19107,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"iLSU-T introduces the first large-scale dataset and baseline benchmark for Uruguayan Sign Language translation, with 185+ hours of interpreted video paired with Spanish text.","keywords":["sign language translation","Uruguayan Sign Language","LSU","multimodal dataset","benchmark","gloss-free translation","video-text alignment"],"falsifier":"Take the over-20-hours of manually aligned episodes, run the automatic clipping rules on the same episodes, and measure the temporal overlap between each automatically produced clip and the expert-marked phrase boundaries; if a large fraction of clips miss the true signing interval, for instance by overlapping it by less than half, then the clip-text pairing that all translation scores depend on is not valid.","tokens_in":15059,"feed_emoji":"🤟","tokens_out":8794,"duration_ms":86292,"temperature":0.7,"pith_summary":"The paper aims to establish iLSU-T as the first large-scale resource for automatic translation of Uruguayan Sign Language (LSU). It assembles more than 185 hours of interpreted television and parliamentary video from 18 professional signers, with Spanish text obtained from the audio track, and runs three state-of-the-art sign language translation systems on it for the first time. The authors argue that sign languages are local, so translation tools need local data, and iLSU-T supplies the missing foundation for building and evaluating LSU translation systems. If the dataset and its alignment pipeline hold up, the reported BLEU and ROUGE scores become the reference numbers that future LSU translation work will be measured against.","feed_headline":"First benchmark set for Uruguayan Sign Language translation","feed_subtitle":"A new open dataset pairs 185 hours of interpreted video with Spanish text and tests three translation models.","key_machinery":"The load-bearing mechanism is a five-stage processing pipeline: region-of-interest identification, signer recognition with a KNN-based face classifier, automatic captioning with the WhisperX speech recognizer, manual phrase alignment by a sign language expert using pauses and epenthesis (transitional movements between signs) as cut points, and linguistic context labeling. For the benchmark experiments, video clips are generated from text phrases by automatic clipping with random pre- and post-delays drawn from uniform distributions, $t_1 \\sim U(0.4,1.2)$ seconds and $t_2 \\sim U(2.1,2.9)$ seconds, chosen by visual inspection. This clipping procedure is what pairs each video clip with a Spanish sentence, so the entire translation benchmark rests on it.","core_discovery":"The central claim is that iLSU-T is the first dataset that makes Uruguayan Sign Language translation tractable: it contains more than 185 hours of curated RGB video of interpreted LSU, synchronized audio, and Spanish text transcriptions, organized into 571 episodes and 86,550 video clips across three broadcast sources. The paper further claims that the dataset supports the first recorded benchmark of state-of-the-art sign language translation methods on LSU, with the parliamentary source giving the best scores (BLEU-4 up to 3.82, ROUGE-L up to 16.05) and the other sources scoring lower. The authors present these numbers not as a solved task but as a baseline that shows both the usefulness of the dataset and the difficulty of aligning interpreted video with text.","pith_inferences":["The random-delay clipping strategy likely understates what an alignment-aware model could achieve; the manual alignments could be used to train a predictor of true phrase boundaries, a test the paper does not run.","The three sources differ in setting, interpreter style, and text duplication, so iLSU-T can serve as a domain-shift benchmark for sign language translation rather than only a single-dataset benchmark.","Because video clips overlap and each phrase is treated independently, the benchmark ignores cross-phrase context such as coreference, so a model using episode-level context might score higher than these isolated-phrase baselines.","If interpreter delay varies by source, per-source clipping parameters would likely change alignment quality; that hypothesis is directly testable with the existing manual annotations."],"forward_implications":["The reported BLEU and ROUGE scores become the first quantitative reference points for LSU translation, allowing future systems to be compared against a fixed benchmark.","Because the parliamentary source's better scores are tied to duplicated phrases between train and test, evaluation on iLSU-T should report duplicate and non-duplicate performance separately.","The manual phrase-alignment annotations covering over 20 hours provide ground truth that can be used to train or validate automatic alignment methods.","The three tested methods are all visual-only and gloss-free, so iLSU-T is a testbed for approaches that add audio, skeleton data, or explicit alignment supervision."],"supporting_citations":[{"why":"Supplies the random-delay clipping methodology that iLSU-T adapts to pair video clips with text phrases.","marker":"[12]"},{"why":"Produces the Spanish sentence transcriptions with word-level timestamps used as the text track.","marker":"[2]"},{"why":"Provides the episode concept and topic labeling approach used to structure iLSU-T.","marker":"[1]"},{"why":"One of the three sign language translation systems benchmarked on the dataset.","marker":"[9]"},{"why":"One of the three sign language translation systems benchmarked on the dataset.","marker":"[39]"},{"why":"One of the three sign language translation systems benchmarked on the dataset; its sentence-embedding similarity idea is adapted with word encoding.","marker":"[41]"},{"why":"Supplies the OpenASL comparison and the duplicate-phrase analysis that explains the parliamentary source's higher scores.","marker":"[35]"},{"why":"Supplies the How2Sign comparison point for judging how hard the iLSU-T benchmark is.","marker":"[17]"},{"why":"Provides the pre-trained I3D-ASL2k visual feature extractor used in the baseline experiments.","marker":"[24]"},{"why":"Provides the pre-trained I3D-BSL5k visual feature extractor used in the baseline experiments.","marker":"[38]"}],"fun_headline_variants":["Uruguayan sign language translation gets its first dataset","185 hours of signed video power new translation benchmark","Open dataset enables first benchmark for Uruguayan sign language","New dataset pairs sign language video with Spanish text","First open benchmark for Uruguayan sign language translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark depends on the assumption that clipping each text-phrase time span with random delays of $t_1 \\sim U(0.4,1.2)$ seconds before and $t_2 \\sim U(2.1,2.9)$ seconds after reliably captures the sign-language rendering of that phrase; if the true interpreter delay falls outside these ranges, or varies by source or signer, the video clips and Spanish sentences are misaligned.","fun_headline_variants_meta":{"raw":{"variants":["Uruguayan sign language translation gets its first dataset","185 hours of signed video power new translation benchmark","Open dataset enables first benchmark for Uruguayan sign language","New dataset pairs sign language video with Spanish text","First open benchmark for Uruguayan sign language translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1276,"prompt_tokens":871,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":487,"tokens_out":405,"duration_ms":4641,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:27:04.047763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the over-20-hours of manually aligned episodes, run the automatic clipping rules on the same episodes, and measure the temporal overlap between each automatically produced clip and the expert-marked phrase boundaries; if a large fraction of clips miss the true signing interval, for instance by overlapping it by less than half, then the clip-text pairing that all translation scores depend on is not valid.","supporting_citations":[{"cited_title":"Dal Bianco, G","cited_arxiv_id":null,"evidence_quote":"Supplies the random-delay clipping methodology that iLSU-T adapts to pair video clips with text phrases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Produces the Spanish sentence transcriptions with word-level timestamps used as the text track."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the three sign language translation systems benchmarked on the dataset."},{"cited_title":"V oskou, K","cited_arxiv_id":null,"evidence_quote":"One of the three sign language translation systems benchmarked on the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the three sign language translation systems benchmarked on the dataset; its sentence-embedding similarity idea is adapted with word encoding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OpenASL comparison and the duplicate-phrase analysis that explains the parliamentary source's higher scores."},{"cited_title":"Duarte, S","cited_arxiv_id":null,"evidence_quote":"Supplies the How2Sign comparison point for judging how hard the iLSU-T benchmark is."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained I3D-ASL2k visual feature extractor used in the baseline experiments."},{"cited_title":"Varol, L","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained I3D-BSL5k visual feature extractor used in the baseline experiments."}],"review_version":1}