{"id":"bbeb24e4-8b73-4bb9-8e2c-4a9c14404337","arxiv_id":"2506.21813","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CAT-SG is a new cataract surgery scene graph dataset with 1.811 million relation annotations, a two-class technique recognition task, and a query-based scene graph generation baseline.","lead":"This paper presents CAT-SG, a large annotated dataset of scene graphs for cataract surgery videos, capturing over 1.8 million tool-tissue relations. It also introduces CatSGG, a model that generates these graphs and reports higher macro F1 than a prior surgical scene graph method.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Close to' relation (~93% of CAT-SG's 1.8M relations) appears derived from CaDIS pseudo-masks rather than manual annotation; if so, the scale claim and CatSGG's geometric evaluation are circular and unreliable.","rationale":"I focused on the provenance and reliability of the dominant 'Close to' relation because it is the largest component of the dataset (92.6% of relations) and the only relation that CatSGG generates geometrically from segmentation masks. The paper's own description in Section 2.2 introduces pseudo-labelled CaDIS masks for object grounding but leaves the generation of 'Close to' unspecified. If those relations are auto-generated from the same pseudo-masks used to supervise M2F, then the benchmark compares a deterministic adjacency computation against its own training signal; the high 'Close to' F1 and a significant part of the macro-F1 advantage over ORacle would be artifacts. Even in the more favorable reading (manual 'Close to' annotations), the pseudo-mask grounding still determines node positions/sizes that propagate into GNN baselines and spatio-temporal reasoning, and no validation of these masks is provided. I therefore regard this as the load-bearing assumption: the dataset's scale, the quality of node grounding, and the validity of the SGG evaluation all depend on it. The reader's verdict (CONDITIONAL) already captures the need for release and reliability analysis; my concern reinforces that condition and adds the specific requirement to clarify and, if necessary, fix the 'Close to' generation. A manual re-annotation of a small random subset, combined with inspection of the generation script, would settle whether the circularity exists and whether the scale claim is honest. Agreement with the reader is partial: we share the pseudo-label vulnerability but I sharpen it into a potential evaluation circularity.","tokens_in":7585,"tokens_out":10529,"duration_ms":102975,"concrete_test":"After CAT-SG is released, inspect the label-generation script to determine whether 'Close to' relations are computed by mask adjacency on the CaDIS pseudo-masks. Independently, manually annotate a 500-frame random subset (object boxes and 'Close to' pairs) and compare with the published CAT-SG labels, reporting 'Close to' precision/recall and box IoU. If 'Close to' precision is below ~90%, or if the generation script derives GT 'Close to' from the same pseudo-masks used to train M2F, the dataset-scale and CatSGG-superiority claims are materially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 states that pseudo-labelled CaDIS masks provide grounding (position, size, bounding box) for objects, but does not specify how the 'Close to' relation (1,677,724 of 1,811,252 samples, Table 1) was generated. Section 3 defines the model's 'Close to' operationally as mask adjacency from M2F predictions. If the ground-truth 'Close to' relations were also computed from the same CaDIS pseudo-masks, the evaluation is circular: M2F is trained to reproduce those masks (mIoU 92.12%), and the geometric relation is an adjacency test on predicted versus pseudo-mask-derived GT. That would inflate CatSGG+'s 'Close to' F1 (91.63) and contribute to the macro-F1 gap over ORacle (43.11 vs 34.65). Even if GT 'Close to' was manually annotated, the pseudo-mask grounding itself is unvalidated: no IoU against manual segmentation is reported, so node positions/sizes and any mask-derived relations could be systematically wrong. The paper's central contributions — the dataset's scale and the CatSGG benchmark — therefore rest on an unclarified, potentially circular use of pseudo-labels.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CAT-SG, a dynamic scene graph dataset for cataract surgery built on the CATARACTS corpus. It contains 29 object classes and 1,811,252 relation instances over 164,162 frames, including eight relation types, a dominant geometric 'Close to' relation, and per-video surgical technique labels. The authors also propose CatSGG/CatSGG+, a Mask2Former-based scene graph generator that uses query embeddings for pairwise relation prediction, and evaluate it against the ORacle baseline. The central claims are that CAT-SG is the first and largest cataract surgery scene graph dataset and that CatSGG+ outperforms prior state of the art on scene graph generation, with additional benchmarks for phase recognition and surgical technique recognition.","tokens_in":7845,"tokens_out":5110,"duration_ms":53228,"significance":"If the annotations are trustworthy, CAT-SG addresses a genuine gap: existing surgical video datasets focus on tools or phases, and a fine-grained dynamic scene graph dataset could support workflow analysis, training feedback, and explainable intraoperative decision support. The scale is notable, and the introduction of surgical technique recognition as a downstream task is a useful addition. The model design is reasonable and the comparison to ORacle is appropriate in spirit. However, the paper's main value depends on the reliability of the pseudo-mask grounding and on the provenance of the 'Close to' relation, which together account for the majority of the dataset and for the reported performance gap. These issues must be resolved before the dataset and benchmark can be considered scientifically sound.","major_comments":[{"comment":"The manuscript never specifies how the 1,677,724 'Close to' ground-truth relations were produced. Section 2.2 says that pseudo-labelled CaDIS masks provide grounding (position, size, bounding box), while Section 3 defines the predicted 'Close to' relation as mask adjacency from M2F predictions. If the ground-truth 'Close to' labels were also derived from the same CaDIS pseudo-masks, then the evaluation of this relation is circular because M2F is trained on those masks; the reported Close-to F1 of 91.63 and the macro-F1 gap over ORacle (43.11 vs 34.65) would be inflated, especially since 'Close to' constitutes roughly 93% of all relations. Please state the exact generation procedure for 'Close to' labels, and if they are mask-derived, validate the pseudo-masks against manual segmentation (e.g., IoU) and re-evaluate the benchmark under a non-circular protocol.","section":"Section 2.2, Table 1, Section 3"},{"comment":"The annotation process is described only in terms of annotator counts and total hours; no inter-annotator agreement measure (e.g., Cohen's kappa or pairwise F1) is reported for relation labels, object presence, or technique labels. For a dataset paper whose central contribution is fine-grained manual annotation, the absence of any agreement measure makes it impossible to assess label noise and the practical upper bound of the downstream benchmarks.","section":"Section 2.2"},{"comment":"The surgical technique recognition task is described as a new downstream task, but the evaluation protocol is missing. With 50 videos and two technique classes, the paper does not state the train/validation/test split, the number of test videos, the cross-validation scheme, or how the mean and standard deviation in Table 5 were computed. As written, the technique recognition results are not reproducible and do not support the claim that CAT-SG enables this task.","section":"Sections 4 and 5, Table 5"},{"comment":"The comparison with ORacle is not sufficiently controlled. The paper says that ORacle's single-view variant was trained and evaluated, but it does not report the used temporal windows, number of training chunks, learning rate, or whether the same splits and frames were used as for CatSGG/CatSGG+. Because the headline claim is that CatSGG+ outperforms ORacle, these training and evaluation details must be specified for the comparison to be meaningful.","section":"Section 4, Table 3"}],"minor_comments":[{"comment":"CatSGG+ reports Pushing F1 = 0.00 while CatSGG reports 4.32, yet the text states that CatSGG+ 'further enhances temporal consistency.' This failure on one relation class should be discussed, because it tempers the temporal-consistency claim and may point to a sampling or class-imbalance issue.","section":"Table 3"},{"comment":"The paper says '8 categories' of interactions and Table 1 lists eight relation rows, but the text in Section 3 and later refers to 'semantic relations' and 'geometric relations' without a clear formal distinction; please define which relations are semantic and which are geometric in the dataset itself.","section":"Section 2.2"},{"comment":"The 'none' class in Table 3 is not defined in the dataset description; please state how negative pairs were sampled and how the 'none' class was constructed for evaluation.","section":"Section 4"},{"comment":"The paper does not provide a dataset release URL, code repository, or license information; for a dataset and benchmark paper these should be included.","section":"General"},{"comment":"The technique recognition experiments include only GATv2; the text says GNNs and LLMs are both baselines, but no LLM result is reported for this task. Please either add the result or remove the claim.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the provenance of the 'Close to' relation and the reliance on unvalidated CaDIS pseudo-masks. If the ground-truth 'Close to' labels are indeed mask-derived, the dataset's scale and the model comparison would need substantial reworking. I therefore recommend that the editor require the authors to clarify the annotation generation procedure and provide a non-circular evaluation before considering the paper for publication. The paper also lacks dataset release details, which is a practical concern for a dataset contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about CAT-SG is that it is exactly what it says on the tin: a large, manually annotated scene graph dataset for cataract surgery, derived from CATARACTS, with over 1.8M relation instances and a new surgical technique recognition task. That is a real contribution to surgical video understanding. The annotation campaign is substantial (9 annotators, 1200+ hours), and the paper ships benchmarks for scene graph generation, phase recognition, and technique recognition. The CatSGG/CatSGG+ model is a reasonable adaptation of existing query-based SGG methods to the surgical setting, and the reported macro-F1 gap over ORacle (43.11 vs 34.65) is credible if the evaluation holds up.\n\nThe soft spots are in the evaluation and in the 'Close to' relation. Over 92% of the 1.8M relations are 'Close to' (1,677,724 of 1,811,252). The paper says the pseudo-labelled CaDIS masks provide grounding (position, size, bounding box), and the model infers 'Close to' from mask adjacency. It never states whether ground-truth 'Close to' was manually annotated or computed from the same pseudo-masks. If the latter, the evaluation is circular: Mask2Former was trained to reproduce those masks (mIoU 92.12) and the 'Close to' F1 of 91.63 becomes a measure of mask reproducibility, not of relationship understanding. That would inflate the macro-F1 gap over ORacle, whose 'Close to' is only 67.03. Even if the GT was manual, the pseudo-mask grounding is unvalidated: no IoU against manual segmentation is reported, so node positions and any mask-derived relations could be systematically wrong.\n\nOther soft spots: there is no inter-annotator agreement measure, no error bars for the SGG results (the technique recognition table has them), and the technique recognition task is binary on 50 videos with no evaluation protocol described—train/test split or cross-validation is not stated. The paper also does not release the dataset or code, which matters for a dataset paper. The phase recognition results do suggest a genuine benefit from the semantic relations (GATv2 beats the prior baseline by 5 p.p.), so the core idea has substance.\n\nWho is this for? Anyone working on surgical video understanding, particularly on fine-grained workflow analysis or technique recognition. It deserves a serious referee, but the authors must clarify how 'Close to' was generated and add basic reliability/reproducibility information before the scale claim can be trusted.","headline":"CAT-SG is a genuinely useful surgical scene graph dataset with real annotation effort, but the 'Close to' relation likely derived from pseudo-masks makes the headline scale and top-line F1 gains unreliable until clarified.","tokens_in":8363,"tokens_out":1852,"would_cite":true,"duration_ms":17645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new cataract surgery scene graph dataset, CAT-SG, holds over 1.8 million annotated relations, and the CatSGG+ model outperforms the prior state of the art on generating such graphs.","keywords":["scene graphs","cataract surgery","surgical data science","dataset","scene graph generation","tool-tissue interaction","surgical workflow recognition","video understanding"],"falsifier":"Manually segment a random sample of CAT-SG frames and compare the masks with the released pseudo-masks; if overlap is low, or if recomputing 'Close to' edges from the manual masks materially changes the relation distribution or downstream F1, the geometric grounding claim would be falsified.","tokens_in":1739,"feed_emoji":"👁️","tokens_out":3759,"duration_ms":103272,"temperature":0.7,"pith_summary":"This paper introduces CAT-SG, a scene graph dataset for cataract surgery built by re-annotating the CATARACTS video collection with fine-grained relations between surgical tools, anatomical structures, and their interactions. The authors claim CAT-SG is the first dataset of its kind, containing over 1.8 million annotated relations across 164,162 frames and adding a new surgical technique recognition task alongside phase recognition. Their CatSGG+ model, which uses a video-pretrained segmentation backbone and pooled query embeddings, achieves a macro F1 of 43.11 on scene graph generation, beating the ORacle baseline at 34.65. The paper also shows that the scene graph annotations improve phase recognition over a baseline that uses only tool detections. If the claims hold, structured scene graphs could support explainable, context-aware analysis of cataract surgery for training and decision support.","feed_headline":"Scene graphs map 1.8M relations in cataract surgery","feed_subtitle":"New dataset links surgical tools to tissue over time; CatSGG+ beats prior best by 8 F1 points.","key_machinery":"The machinery that carries the argument is the scene graph plus the CatSGG+ architecture. Nodes are surgical instruments or anatomical structures grounded by position, size, and bounding box; edges are relations, with semantic relations predicted from pair proposals built by concatenating two instance query vectors from Mask2Former, $pair_n = [q_i; q_j]$, and the geometric relation 'Close to' derived from adjacent predicted masks. CatSGG+ adds temporal context by max-pooling same-class query vectors across the eight frames in a chunk before building pair embeddings, so the relation predictor sees motion and appearance change, not just a single frame. The relation existence model uses a 2-layer network with sigmoid, and the relation type model is a 3-layer multi-label classifier, trained with binary cross-entropy.","core_discovery":"The central claim is that dynamic scene graphs can represent cataract surgery at scale and that this representation enables fine-grained, temporally aware surgical workflow understanding. CAT-SG annotates 29 object classes and eight semantic relations—holding, activation, pushing, pulling, cutting, inserting, retracting—plus the geometric relation 'Close to', totaling 1,811,252 relations on 164,162 frames sampled at 5 fps from 50 CATARACTS videos. The authors report that CatSGG+, their scene graph generation model built on Mask2Former with a VideoSwin backbone pretrained on surgical videos and with max-pooled query embeddings across 8-frame chunks, reaches macro F1 43.11, outperforming ORacle's 34.65. They further report that using semantic relations in a GATv2 phase recognizer raises 30-frame-window accuracy from 73.77 to 78.63, and that their technique recognition baseline reaches 68.75 accuracy for distinguishing 'Stop and Chop' from 'Divide and Conquer' using 10-second windows at 5 fps.","pith_inferences":["Editorial inference: because 'Close to' accounts for about 93% of all relations, reported scene graph F1 is dominated by a binary adjacency task; a useful stress test would be to evaluate CatSGG+ on the seven semantic relations alone, excluding 'Close to'.","Editorial inference: the geometric grounding depends on pseudo-labelled CaDIS masks, so the dataset's spatial fidelity is only as good as those masks; comparing a manually segmented subset against the pseudo-masks would quantify how much mask error propagates into node grounding and 'Close to' relations.","Editorial inference: the same annotation recipe could transfer to other surgical video datasets that have phase labels but no interaction annotations, since scene graph structure is not specific to cataract surgery.","Editorial inference: CatSGG+ max-pools same-class queries across an 8-frame chunk; attention-based pooling or variable-length chunks might better capture short-lived interactions such as 'cutting' (1,925 samples), which the current model predicts with lower F1."],"forward_implications":["CAT-SG provides 1.8 million relations over 164,162 frames, making it the largest surgical scene graph dataset among comparable efforts and a benchmark for tool–tissue interaction modeling.","The CatSGG+ result of 43.11 macro F1 against ORacle's 34.65 indicates that a segmentation-based, query-pooling approach can outperform a large vision-language model for surgical scene graph generation.","Semantic relations improve phase recognition: GATv2 with CAT-SG relations reaches 78.63 accuracy and 70.15 F1 on a 30-frame (90-second) window, beating the Holm et al. baseline of 73.77 accuracy and 64.93 F1.","The new surgical technique recognition task is learnable at 5 fps: the GATv2 baseline reaches 68.75 accuracy and 48.40 F1, with finer temporal sampling outperforming lower-resolution windows.","Because relations are human-readable, downstream phase and technique predictions built on these graphs can be traced to specific interactions such as 'phaco handpiece activating' or 'cystotome pulling'."],"supporting_citations":[{"why":"Supplies the 50 cataract surgery videos, phase annotations, and frame sampling that CAT-SG extends with scene graph annotations.","marker":"[1]"},{"why":"Provides the video-language pretrained VideoSwin backbone trained on 2,900 YouTube cataract surgery videos, which the segmentation model fine-tunes for CAT-SG.","marker":"[8]"},{"why":"CaDIS is the source of the pseudo-labelled segmentation masks used to ground objects with position, size, and bounding boxes.","marker":"[9]"},{"why":"Mask2Former is the segmentation architecture whose instance query embeddings are reused for relation prediction in CatSGG.","marker":"[4]"},{"why":"ORacle is the prior state-of-the-art surgical scene graph generation model that CatSGG and CatSGG+ are compared against.","marker":"[23]"},{"why":"Provides the dynamic surgical scene graph baseline for phase recognition and the mask-adjacency approach for 'Close to' relations.","marker":"[10]"},{"why":"Supplies the pair-proposal query embedding design and the two-stage relation existence/classification models that CatSGG adapts, including max pooling across frames.","marker":"[20]"}],"fun_headline_variants":["1.8M relations map dynamic scene graphs for cataract surgery","Cataract surgery gets first large-scale dynamic scene graph dataset","Scene graph generation for cataract surgery outperforms prior methods","CAT-SG dataset: 1.8M relations capture surgical workflows","Dynamic scene graphs for fine-grained understanding of cataract surgery"],"cache_read_input_tokens":10496,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-labelled segmentation masks based on CaDIS correctly ground every object, since node positions, sizes, bounding boxes, and the 'Close to' relations computed from mask adjacency all depend on those masks.","fun_headline_variants_meta":{"raw":{"variants":["1.8M relations map dynamic scene graphs for cataract surgery","Cataract surgery gets first large-scale dynamic scene graph dataset","Scene graph generation for cataract surgery outperforms prior methods","CAT-SG dataset: 1.8M relations capture surgical workflows","Dynamic scene graphs for fine-grained understanding of cataract surgery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000921,"raw_usage":{"total_tokens":3938,"prompt_tokens":924,"completion_tokens":3014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2930}},"tokens_in":540,"tokens_out":3014,"duration_ms":24272,"temperature":1.0,"reasoning_tokens":2930,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:17:59.478201+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually segment a random sample of CAT-SG frames and compare the masks with the released pseudo-masks; if overlap is low, or if recomputing 'Close to' edges from the manual masks materially changes the relation distribution or downstream F1, the geometric grounding claim would be falsified.","supporting_citations":[{"cited_title":"Medical Image Analysis 52, 24–41 (2019)","cited_arxiv_id":null,"evidence_quote":"Supplies the 50 cataract surgery videos, phase annotations, and frame sampling that CAT-SG extends with scene graph annotations."},{"cited_title":"Watch and Learn: Leveraging Expert Knowledge and Language for Surgical Video Understanding","cited_arxiv_id":"2503.11392","evidence_quote":"Provides the video-language pretrained VideoSwin backbone trained on 2,900 YouTube cataract surgery videos, which the segmentation model fine-tunes for CAT-SG."},{"cited_title":"CaDIS: Cataract Dataset for Image Segmentation","cited_arxiv_id":"1906.11586","evidence_quote":"CaDIS is the source of the pseudo-labelled segmentation masks used to ground objects with position, size, and bounding boxes."},{"cited_title":"In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention","cited_arxiv_id":null,"evidence_quote":"ORacle is the prior state-of-the-art surgical scene graph generation model that CatSGG and CatSGG+ are compared against."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the pair-proposal query embedding design and the two-stage relation existence/classification models that CatSGG adapts, including max pooling across frames."}],"review_version":1}