{"id":"f8396162-67b3-4736-91e2-4fa826ac46be","arxiv_id":"2412.15054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GIRAFE releases 65 color high-speed laryngeal videos, 760 manual glottal gap masks, automatic segmentation baselines, and facilitative playbacks.","lead":"GIRAFE is a public repository of 65 high-speed videos of the larynx with expert-created masks for the glottal gap. It is intended to give voice researchers a shared benchmark for testing automatic segmentation algorithms and generating facilitative playbacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's stated 65-video manual-annotation coverage is contradicted by Sections 2.3 and 2.6; if only 38 recordings have masks, the abstract's central claim overstates the resource.","rationale":"The reader correctly identifies the coverage mismatch in the strongest_claim, but chooses single-expert reliability as the weakest assumption. I argue the more load-bearing issue is the internal contradiction about annotation coverage. The central contribution is a resource with manual glottal masks; whether it actually provides masks for all 65 videos is a factual, checkable property, not a matter of statistical convention. The annotation-reliability concern is legitimate and should be acknowledged, but many medical image benchmarks start from one expert with review; this is a limitation, not a contradiction. Conversely, the abstract's 'All of them were manually annotated' is directly false under the 38-video reading. The paper's own metadata schema (the Man flag) provides an easy audit. The DL validation's small test set (4 patients) does not undermine the dataset-release central claim because it is explicitly exemplary. Previous publications using the corpus and public code provide independent support. I therefore recommend keeping the conditional verdict: the resource may be useful, but the annotation-coverage claim must be corrected and audited.","tokens_in":124,"tokens_out":5737,"duration_ms":72692,"concrete_test":"Download the GIRAFE Zenodo package (DOI 10.5281/zenodo.13773163, version 1.0.0). For each of the 65 folders under /Raw_Data, parse metadata.json and count recordings whose 'Man' field is 'Yes'; then count subdirectories under /Seg_FP-Results/Manual_Segmentation and PNG files under /Training/labelsTr. If the three counts are 38, 38, and 760, the abstract's 'All of them' is false and the paper must be corrected to state that manual ground truth covers 38 videos (760 frames), with the remaining 27 carrying only automatic segmentations. If the count is 65, then Sections 2.3 and 2.6 and the 38-subfolder claim are wrong; either way, the discrepancy must be resolved before the dataset description can be trusted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GIRAFE provides 65 recordings, with 'all of them' manually annotated by an expert with glottal-gap masks. Section 2.3 states that 'Manual segmentation was conducted on 38 out of the 65 videos', producing 760 segmented frames. Section 2.6 confirms the structural consequence: the Manual_Segmentation folder contains exactly 38 subfolders. Section 2.5 introduces a per-recording metadata flag 'Man' precisely to indicate which videos have manual masks, but the paper nowhere reports the aggregate count of Man=Yes across all 65 recordings. This discrepancy is not a stylistic issue: it determines what a user actually receives. If 38 is correct, then 27 of the 65 videos contain only automatic/semiautomatic segmentations (InP for all, Loh for 39), and any user expecting supervised ground truth for all 65 has been misled. The annotation-reliability issue is secondary: single-expert masks are common in medical imaging benchmarks and can be reported as a limitation, whereas an internal contradiction about the dataset's own contents affects every downstream use. The concrete check below settles which number is authoritative.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GIRAFE, a publicly available dataset of 65 high-speed videoendoscopic (HSV) recordings of the vocal folds from 50 patients, intended to support semantic segmentation of the glottal gap and evaluation of facilitative playbacks (GAW, GVG, PVG). The repository includes raw AVI videos, JSON metadata, manual segmentations (reported in Section 2.3 as 38 of the 65 videos, yielding 760 frames), automatic/semi-automatic segmentations from the authors' InP method and the Loh method, and the derived playbacks. A technical validation trains UNet and SwinUnetV2 on the manual masks and compares them against InP and Loh on four test patients (20 frames each) in Table 3. The paper positions GIRAFE as a complement to the existing BAGLS dataset, adding color recordings and playback evaluation, and provides open data and code.","tokens_in":16837,"tokens_out":3561,"duration_ms":30968,"significance":"If the stated coverage is corrected, GIRAFE is a useful and reproducible contribution to laryngeal imaging: it is, to the authors' knowledge, the first public color HSV dataset with glottal-gap annotations, and it ships open code, trained baseline models, and a documented directory structure. The dataset has already supported several prior studies, which is a positive indicator of practical utility. The main significance is as a benchmark resource rather than a methodological advance; the DL validation is explicitly illustrative. However, the abstract's claim that all 65 recordings were manually annotated is contradicted by the body of the paper, and this discrepancy directly affects what a user receives and how the dataset's value is advertised. With the correction, the resource is of moderate but real value to the community.","major_comments":[{"comment":"The abstract states that 'All of them were manually annotated by an expert, including the masks corresponding to the semantic segmentation of the glottal gap,' and Section 1 repeats the promise of 'annotations meticulously performed and rigorously reviewed.' This is contradicted by Section 2.3, which says 'Manual segmentation was conducted on 38 out of the 65 videos,' and by Section 2.6, which reports that the Manual_Segmentation folder contains exactly 38 subfolders. The discrepancy is load-bearing because it determines the dataset's actual content: 27 of the 65 recordings lack manual masks and can only be used with the InP (and possibly Loh) segmentations. The abstract and all downstream descriptions must be corrected to state the 38-video coverage, and the dataset documentation should report the aggregate count of the Man=Yes metadata flag so users can verify the coverage without opening each recording.","section":"Abstract; Section 2.3; Section 2.6"},{"comment":"The technical validation uses only four test recordings, each with 20 frames (80 frames total per model), and reports no inter-annotator agreement, confidence intervals, or statistical significance for the DICE/Jaccard metrics in Table 3. The paper explicitly disclaims that the validation is 'not intended to be exhaustive,' but the claim that the dataset provides a 'gold standard' for supervised segmentation is weakened by the absence of any evidence about the reliability of the manual masks. I recommend adding at least a small inter-annotator study (even on a subset) and reporting per-frame variability, or alternatively reframing the manual masks as single-expert annotations and softening the 'gold standard' language throughout the paper.","section":"Section 2.8; Table 3"},{"comment":"The sentence 'These segmentations were further validated using two established image processing techniques: InP [22] and Loh [20]' is misleading. Comparing automatic segmentations (InP, Loh) against the manual masks is an evaluation of those automatic methods, not a validation of the manual segmentations. If the intent is to demonstrate that the manual masks are reliable, the appropriate evidence is clinician re-annotation, inter-annotator agreement, or some other independent verification, none of which is reported. Please rephrase this sentence to describe what was actually done, and avoid implying that agreement between automatic methods and manual masks establishes the correctness of the masks.","section":"Section 2.3"}],"minor_comments":[{"comment":"The phrase 'PKL Phyton ® format' contains a typo: 'Phyton' should be 'Python'.","section":"Section 2.3"},{"comment":"There is a duplicated article in 'analysis of the the vocal folds trajectory'; also, 'glottal axis’s' should be 'glottal axis' or 'glottal-axis' to avoid a possessive apostrophe error.","section":"Section 2.6"},{"comment":"The heading 'Experimental design, Material and Methods' appears mid-paper without a section number and is not listed in the initial section outline; please either format it consistently or integrate it into the preceding subsection.","section":"Section 2.8"},{"comment":"The database tree structure is visually cluttered, especially the parent folder labels for multiple patients, making it difficult to infer the actual directory nesting; a simplified tree or a listing of folder names with the number of entries per folder would be clearer.","section":"Figure 5"},{"comment":"The CRediT statement credits 'D.P.S.' with data collection, but no such author is listed among the manuscript's authors; the contribution should be attributed in the author list or acknowledgments, with the consent of that person if appropriate.","section":"CRediT Author Statement"},{"comment":"The text says the Training directory contains 760 images 'corresponding to the same number of consecutive frames extracted from the original sequences'; please clarify how the 760 frames are distributed across the 38 manually segmented videos (e.g., 20 consecutive frames per video), since the phrasing is ambiguous.","section":"Section 2.7"}],"recommendation":"major_revision","confidential_remarks":"The internal contradiction between the abstract and Section 2.3 is the main correctness issue and must be resolved before acceptance; it is fixable within the scope of a revision. I also note a high rate of self-citation (references 1, 10, 19, 21, 22, 49, 50, 54, 62) and the use of the authors' own InP method as a baseline, which is acceptable but should be made explicit as a potential bias in the comparison. The CRediT statement credits an unlisted individual (D.P.S.) with data collection; this is an authorship/acknowledgment issue for the editor to verify. The dataset's claim to be the first public color HSV dataset should be checked against the actual contents of BAGLS, since the paper relies on BAGLS being predominantly grayscale."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the abstract says all 65 recordings were manually annotated, but Section 2.3 says manual segmentation was done on 38 of the 65 videos (760 frames), and the repository's Manual_Segmentation folder has 38 subfolders. That is an internal contradiction about the actual contents of the resource, not a stylistic slip. Anyone who downloads the dataset expecting supervised ground truth for all 65 recordings will be misled. The paper needs to correct the abstract, report the exact count of Man=Yes videos, and state clearly which recordings have manual masks.\n\nThat said, the resource itself is genuinely new and useful. GIRAFE is the only public color HSV dataset for glottal gap segmentation with manual masks and facilitative playbacks, and it complements BAGLS rather than competing with it. The authors provide a Zenodo DOI, GitHub code, detailed metadata, and a clear directory structure. The inclusion of InP and Loh baselines plus two DL models gives users a starting point for comparison. The documentation of the PKL file formats, FP synthesis scripts, and training splits is thorough. This is a real contribution to a niche but clinically relevant area.\n\nThe softer spots beyond the annotation-count contradiction: the DL validation is genuinely small (four test patients, 20 frames each, no confidence intervals), though the authors explicitly call it a demonstration rather than a full benchmark. Single-expert masks with one verification pass are common in medical imaging, but the paper should report inter-annotator agreement or at least acknowledge this as a limitation. These issues are fixable. The central dataset artifact holds up; the presentation does not.\n\nWho gets value: researchers working on glottal segmentation, laryngeal imaging, or FP extraction, especially those needing color data. It deserves a serious referee. I would send it to peer review with a request for major revision focused on the annotation-coverage discrepancy and a more measured description of what the dataset actually contains.","headline":"A useful new dataset that overstates its own manual-annotation coverage in the abstract; fix that and it's a solid contribution.","tokens_in":17397,"tokens_out":1399,"would_cite":true,"duration_ms":14338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GIRAFE is a publicly released dataset of 65 high-speed color videoendoscopic vocal-fold recordings with expert manual glottal-gap segmentations, automatic segmentation baselines, and facilitative playbacks, designed to fill the gap left…","keywords":["GIRAFE","high-speed videoendoscopy","vocal folds","glottal gap segmentation","semantic segmentation","facilitative playbacks","medical imaging dataset","deep learning"],"falsifier":"Concrete check: have two or more independent experts re-annotate a random subset of the 760 glottal-gap frames and compute inter-annotator DICE agreement; if agreement is low or if the released masks show clear systematic errors, the dataset's validity as a gold standard is undermined. Additionally, a simple audit of the released repository can verify whether manual masks exist for all 65 recordings or only for the 38 stated in the methods section.","tokens_in":16450,"feed_emoji":"🗣️","tokens_out":3587,"duration_ms":22409,"temperature":0.7,"pith_summary":"The paper introduces GIRAFE, a data repository meant to advance the automatic semantic segmentation of the glottal gap in high-speed videoendoscopic recordings of the vocal folds and to support the synthesis and evaluation of facilitative playbacks. It argues that the lack of public annotated datasets in this domain has limited reproducibility and the training of deep-learning models, and that GIRAFE provides a much-needed complement to the existing BAGLS dataset by adding color recordings, manual masks, and playback outputs. The dataset comprises 65 recordings from 50 patients, with manual segmentations covering one complete glottal cycle per annotated recording, and it already has been used in several published studies.","feed_headline":"GIRAFE opens 65 vocal-fold videos with glottal-gap masks","feed_subtitle":"A public color high-speed laryngoscopy benchmark with manual segmentations and facilitative playbacks for training segmentation models.","key_machinery":"The central object is the GIRAFE repository itself: a structured corpus of raw HSV videos (AVI), per-patient JSON metadata, manual and automatic segmentation masks stored as PKL files, facilitative playback outputs, and a training directory with 760 PNG frames and their corresponding labels. The load-bearing element is the expert manual annotation of the glottal gap, which serves as ground truth for all reported DICE, Jaccard, recall, and precision scores, and from which the facilitative playbacks are synthesized.","core_discovery":"On its own terms, GIRAFE establishes a new public benchmark for glottal-gap segmentation: 65 high-speed color videoendoscopic recordings acquired at 4,000 fps from 50 patients, spanning healthy voices, diagnosed voice disorders, and unknown health states. The manual gold standard consists of expert delineations of the glottal gap, verified by an otolaryngologist, produced for 38 of the 65 videos and totaling 760 frames; the abstract of the paper describes all 65 recordings as manually annotated. Alongside the manual masks, the repository provides segmentations from two classical image-processing methods (InP and Loh), deep-learning baselines (UNet and SwinUnetV2), facilitative playbacks (GAW, GVG, PVG, and digital kymograms), and predefined training/validation/test splits with JSON metadata, all released openly.","pith_inferences":["The discrepancy between the abstract's claim that all 65 recordings were manually annotated and the methods section's statement that manual segmentation was performed on 38 recordings should be resolved by users checking the released metadata; if only 38 have manual masks, the effective supervised training set is smaller than the headline number suggests.","Because the ground truth comes from a single expert with no reported inter-annotator agreement, benchmark scores on GIRAFE should be interpreted as reflecting one expert's delineation style; an independent multi-annotator study would materially strengthen the dataset's reliability.","The color versus grayscale advantage could be tested directly by comparing segmentation performance on GIRAFE against BAGLS under matched training conditions, a comparison the paper motivates but does not perform.","The inclusion of facilitative playbacks opens the possibility of treating playback fidelity as a task-specific evaluation metric, which may reward segmentations that preserve temporal and morphological features relevant to clinical assessment."],"forward_implications":["Researchers can train and compare deep-learning glottal segmentation models on a standardized, openly available benchmark with predefined splits, enabling fair and reproducible evaluation.","The color recordings and the inclusion of playback outputs allow downstream evaluation of segmentation quality through facilitative playbacks, not only through pixel-wise metrics.","GIRAFE complements BAGLS by adding color data and playback-based analysis, potentially supporting cross-dataset generalization studies and foundation-model fine-tuning in laryngeal imaging.","The provided baseline results for InP, Loh, UNet, and SwinUnetV2 give immediate reference points for future segmentation methods on this dataset."],"supporting_citations":[{"why":"Describes BAGLS, the only other public glottis-segmentation dataset, which GIRAFE explicitly complements by adding color and playback evaluation.","marker":"[60]"},{"why":"Defines the InP automatic segmentation method applied to the full dataset and used as one baseline in the technical validation.","marker":"[22]"},{"why":"Defines the Loh semiautomatic segmentation method, applied to 39 videos and used as a baseline.","marker":"[20]"},{"why":"Supplies the UNet architecture trained from scratch on the 760 manual annotations, one of the two deep-learning baselines.","marker":"[63]"},{"why":"Supplies the SwinUnetV2 architecture trained as the second deep-learning baseline.","marker":"[64]"},{"why":"Provides the GlottalImageExplorer toolbox used to compute the Loh segmentations and trajectories.","marker":"[68]"},{"why":"Zenodo repository identifier for the GIRAFE dataset itself, confirming its public availability.","marker":"[70]"}],"fun_headline_variants":["GIRAFE dataset: 65 vocal-fold videos with glottal-gap masks","New public benchmark for glottal-gap segmentation from high-speed laryngoscopy","GIRAFE releases expert-annotated glottal-gap masks for 65 laryngoscopy videos","First public high-speed laryngoscopy dataset with semantic glottal-gap masks","Open dataset for automatic glottal-gap segmentation from vocal-fold video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single expert's manual delineations of the glottal gap are an accurate gold standard; since no inter-annotator agreement is reported and only one otolaryngologist verification pass is mentioned, any systematic bias in those masks would propagate into every reported segmentation score and into the dataset's value as a benchmark.","fun_headline_variants_meta":{"raw":{"variants":["GIRAFE dataset: 65 vocal-fold videos with glottal-gap masks","New public benchmark for glottal-gap segmentation from high-speed laryngoscopy","GIRAFE releases expert-annotated glottal-gap masks for 65 laryngoscopy videos","First public high-speed laryngoscopy dataset with semantic glottal-gap masks","Open dataset for automatic glottal-gap segmentation from vocal-fold video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1318,"prompt_tokens":995,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":611,"tokens_out":323,"duration_ms":3311,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:39:43.196731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Concrete check: have two or more independent experts re-annotate a random subset of the 760 glottal-gap frames and compute inter-annotator DICE agreement; if agreement is low or if the released masks show clear systematic errors, the dataset's validity as a gold standard is undermined. Additionally, a simple audit of the released repository can verify whether manual masks exist for all 65 recordings or only for the 38 stated in the methods section.","supporting_citations":[{"cited_title":"Andrade-Miranda, J","cited_arxiv_id":null,"evidence_quote":"Defines the InP automatic segmentation method applied to the full dataset and used as one baseline in the technical validation."},{"cited_title":"Lohscheller, H","cited_arxiv_id":null,"evidence_quote":"Defines the Loh semiautomatic segmentation method, applied to 39 videos and used as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SwinUnetV2 architecture trained as the second deep-learning baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GlottalImageExplorer toolbox used to compute the Loh segmentations and trajectories."},{"cited_title":"Andrade-Miranda, D","cited_arxiv_id":null,"evidence_quote":"Zenodo repository identifier for the GIRAFE dataset itself, confirming its public availability."}],"review_version":1}