{"id":"27d873dc-8438-47b0-9484-639b5d99acaf","arxiv_id":"2501.01685","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors release three RGB-D instance segmentation benchmarks and an intra-modal attention fusion module that modestly improves segmentation in most, but not all, tested configurations.","lead":"This paper introduces three RGB-D instance segmentation benchmarks, two derived from existing indoor semantic datasets and one newly captured box-pile dataset, and proposes an attention-based module for fusing color and depth features. The module gives modest accuracy gains in most settings, but the claim of consistent improvement is not fully supported by the reported tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark masks are unvalidated automatic conversions of semantic labels, so the benchmark claim is unsupported; Table 4 separately contradicts the 'consistent gains' claim for IAM.","rationale":"The reader's weakest assumption is exactly the most load-bearing issue: instance masks for NYUDv2-IS and SUN-RGBD-IS are derived automatically from semantic segmentation labels without human verification. My read agrees that this threatens the benchmark claim more than any other single issue, because all reported method comparisons are evaluated against those masks. The paper does provide public dataset URLs and a clear specification of categories and splits, which is useful, but those resources do not substitute for validation of the instance labels themselves. The consistency problem in Table 4 is a real, directly visible contradiction of the 'consistently outperforms' wording, and it should be corrected by softening the claim or by showing a setting where the comparison is actually consistent. However, that issue is local and reparable, whereas the benchmark-label concern is foundational. Since the reader already reached a conditional verdict on this basis, my stress test does not move the verdict. If the proposed spot-check shows severe merges or splits, the verdict should become reject because the central dataset contribution and all derived numbers would be invalid; if the spot-check passes, the conditional can be relaxed after the authors fix the overstated consistency claim and report error bars or multiple seeds.","tokens_in":15276,"tokens_out":5789,"duration_ms":63066,"concrete_test":"Download the released NYUDv2-IS and SUN-RGBD-IS annotations and sample about 50 images per dataset stratified by object category. Have at least two independent annotators draw instance masks for those images, then compute per-image instance-count agreement and mask IoU between the automatic labels and the human labels. If mean mask IoU falls below 0.8 or more than 10% of sampled images show an instance-count mismatch (merges or splits), the connected-component-derived masks cannot be treated as ground truth for instance segmentation, and the conditional verdict should move to reject; if human agreement is high, the benchmark concern is resolved while the Table 4 wording still needs correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing condition for the benchmark half of the paper is that NYUDv2-IS and SUN-RGBD-IS contain correct ground-truth instances. Section 3.1 describes only an automatic conversion: instance masks are 'generated' from the original object class annotations via 'extracting binary instance masks', with no step that distinguishes distinct objects of the same class and no human verification, inter-annotator agreement, or quality control. The Introduction's 're-annotating' (Section 1) is not substantiated by that pipeline. Because semantic labels partition pixels by class, the only available procedure for producing instance masks without additional annotation is connected-component grouping of each class mask. That procedure merges adjacent same-class objects (touching chairs, stacked items, tableware clusters) and can split a single object across occluders, so instance counts and boundaries can deviate systematically from true object instances. Since every AP number in Tables 3-5 is computed against these unvalidated masks, the reusable-resource contribution and all derived comparisons rest on an unverified assumption. Independently, even taking Table 4 at face value, the 'consistently outperforms existing fusion methods' claim is false as written: DETR on SUN-RGBD-IS gives Ours 22.9 AP^seg, below Intra 23.6 and Inter 23.4. The benchmark-validity issue is more load-bearing because it undermines the central dataset contribution and every evaluation built on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces three RGB-D instance segmentation benchmarks (NYUDv2-IS, SUN-RGBD-IS, and Box-IS) and a fusion module, Intra-modal Attention Mix (IAM), inserted into two-stream DETR and SOLQ architectures. The first two benchmarks are derived from existing semantic-segmentation datasets via COCO-format conversion, while the third is newly captured with an Intel RealSense D455. The paper reports instance-segmentation AP comparisons against early/late fusion, intra/inter-modal attention, SA-Gate, and CMX, together with ablations on block placement, feature-integration design, and depth-sensor quality. The two central claims are that the derived datasets are valid instance-level benchmark resources and that IAM consistently outperforms existing RGB-D fusion methods.","tokens_in":15564,"tokens_out":7097,"duration_ms":66891,"significance":"If the benchmark masks are valid, the datasets would fill a genuine gap: public real-world RGB-D instance segmentation benchmarks are scarce, and the paper provides statistics, splits, and public links for all three datasets. The method side is modest but useful: IAM is a light-weight module that improves SOLQ in most reported settings, and the paper includes multiple baselines and ablations. However, the significance of the benchmark contribution is conditional on the unverified instance-mask construction described in Section 3.1, and the 'consistent improvement' claim is contradicted by Table 4. The absence of any human verification or inter-annotator agreement for the derived instance masks is the main correctness risk for the dataset half of the paper.","major_comments":[{"comment":"The benchmark construction does not establish that the derived masks are true object instances. The text states only that instance masks were generated from the original object-class annotations by 'extracting binary instance masks' and converting them to COCO-style polygons; there is no step that separates distinct touching objects of the same class, no handling of occluders that split a single object, and no human verification or inter-annotator agreement. Since every AP number in Tables 3-5 is computed against these masks, the reusable-benchmark contribution and all derived comparisons rest on unvalidated ground truth. The Introduction's claim of 're-annotating' (Section 1) is not substantiated by the described pipeline. The authors should provide a validation study, per-image instance-count comparisons, or use existing genuinely instance-level annotations before the benchmarks can be accepted as instance segmentation resources.","section":"Section 3.1"},{"comment":"The claim that 'models using our IAM module consistently outperformed' intra- and inter-modal attention is contradicted by the DETR rows on SUN-RGBD-IS: Ours achieves APseg = 22.9, while Intra achieves 23.6 and Inter achieves 23.4; Ours is also lower in APdet (29.3 vs. 30.2 and 30.1). The paper must either scope the consistency claim to SOLQ or to the datasets where it holds, and it should analyze the negative DETR result instead of asserting universal superiority. This is load-bearing because the abstract, introduction, and Section 5.3 all advertise consistent gains over fusion baselines.","section":"Section 5.3, Table 4"},{"comment":"The ablation baseline is ambiguous and inconsistent with Table 3. Table 6 reports 'Backbone' APseg = 28.6, which matches the 'Early' fusion result in Table 3 (28.6), not the RGB-only SOLQ result (33.1). Table 8 similarly uses SOLQ = 28.6 as the starting point. The 7.2% gain attributed to IAM+CDF is therefore a gain over early fusion, not over the RGB-only or late-fusion baselines that the text elsewhere compares against. The ablation protocol should be aligned with the comparison protocol in Table 3, or the text should explicitly define which baseline is being augmented.","section":"Section 5.4.1 and 5.4.3, Tables 6 and 8"},{"comment":"The claimed 'mixup-inspired' mechanism is not reflected in the algebra. Expanding the block matrices yields QK^T = Qrgb Krgb^T + Qd Kd^T, which is a block-diagonal attention with no cross-modal terms; this is equivalent to computing attention within each modality separately and summing the results, not to mixing or blending features across modalities. If the intended operation is exactly this per-modality attention sum, the mixup analogy should be removed or substantially qualified, because the current description overstates the conceptual novelty of the module.","section":"Section 4.2.1, Eq. (7)"}],"minor_comments":[{"comment":"The phrase 'we construct two public RGB-D instance segmentation dataset' should be 'datasets', and the sentence should be reworded for grammatical consistency.","section":"Section 1, contribution bullet"},{"comment":"The paper provides public repository links, but it does not state the licenses, file formats, or evaluation scripts included in those repositories; adding this information would improve reproducibility.","section":"Section 3.1"},{"comment":"The shape and broadcasting of Wn are not specified: after global average pooling and sigmoid, Wn appears to be a per-channel vector, but the text should state its dimensionality and how it is applied to Frgb and Fd in Eqs. (11) and (12).","section":"Section 4.2.2, Eqs. (10)-(12)"},{"comment":"Box-IS results are reported only for SOLQ, with no DETR table or a clear statement that DETR was omitted due to the difficulty described in the text; please include the DETR numbers or an explicit note on their absence.","section":"Section 5.3, Table 5"},{"comment":"Reference [6] lists the venue as 'Winter Conf. on Application of Computer Vision (ECCV)'; the correct venue is WACV, and the citation should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark-validity issue is the main risk for acceptance: the paper presents NYUDv2-IS and SUN-RGBD-IS as re-annotated instance benchmarks, but Section 3.1 describes only an automatic conversion from semantic labels with no validation. If the authors cannot provide a convincing validation study or replace the masks with genuine instance annotations, I would not support acceptance. The Table 4 contradiction is more localized and can be fixed by scoping the claims, but it suggests that the abstract and Section 5.3 need careful rewriting. I recommend asking for a revised version with the above points addressed before making a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful dataset paper wrapped around a modest method. The three RGB-D instance segmentation benchmarks are new artifacts, and Box-IS fills a real gap for robotic box depalletizing. The GitHub releases make the resources usable. The IAM module is a minor variant of existing attention fusion, with a mixup-inspired intra-modal term plus a channel-wise gate; nothing wrong with that, but it is not a conceptual leap.\n\nThe soft spots are where the reader's stress-test lands. First, the conversion of NYUDv2 and SUN-RGBD semantic labels into instance masks is described in one sentence ('extracting binary instance masks') with no detail on how same-class touching objects are separated, no human verification, no inter-annotator agreement, and no error bars. That is load-bearing because every AP number in the paper is computed against those masks. The intro says 're-annotating,' but the methods section does not substantiate it. Second, Table 4 directly contradicts the 'consistent improvements over existing fusion methods' claim: DETR on SUN-RGBD-IS gives Ours 22.9 APseg versus Intra 23.6 and Inter 23.4. The text's hand-wave about DETR's architecture does not fix the inconsistency. On Box-IS, the SOLQ gain over late fusion is 0.5 AP, and over RGB it is 0.4, so the 'notable enhancements' are sometimes marginal.\n\nI would push back mildly on the reader's circularity score: there is no circularity loop here, just a data provenance issue, and the method's parameters are learned from the training split. That worry is overblown.\n\nWho this is for: people building RGB-D instance segmentation baselines and robotics manipulation researchers will get value from the released benchmarks, if the masks hold up. The method section is a reasonable baseline for future fusion work. It deserves a serious referee, but the authors should be pushed to (a) document the mask-generation pipeline precisely, ideally with human verification or at least a random-sample audit, (b) fix the 'consistently outperforms' claim or the table, and (c) release code and error bars. If those are addressed, it is a publishable contribution; as is, the central claims are not fully supported.","headline":"The new benchmarks are the real contribution, but unvalidated mask conversion and one contradictory table undercut the central claims; worth a serious referee, not a desk reject.","tokens_in":16072,"tokens_out":2317,"would_cite":false,"duration_ms":21684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple intra-modal attention mixing module, evaluated on three new RGB-D instance segmentation benchmarks, consistently outperforms prior fusion methods such as early fusion, late fusion, and attention-based approaches.","keywords":["RGB-D instance segmentation","benchmark dataset","multimodal fusion","intra-modal attention","attention mechanism","depth sensing","indoor scene understanding","DETR"],"falsifier":"Take a random sample of images from NYUDv2-IS and SUN-RGBD-IS, have annotators draw instance masks by hand, and compute agreement with the connected-component-derived masks. If a substantial fraction of instances (for example more than 10%) differ in whether they are merged or split, the benchmarks would not measure what they claim, and model rankings on them could be reversed.","tokens_in":15097,"feed_emoji":"🎯","tokens_out":7850,"duration_ms":62962,"temperature":0.7,"pith_summary":"This paper introduces three RGB-D instance segmentation benchmarks — NYUDv2-IS, SUN-RGBD-IS, and Box-IS — and a lightweight fusion module called Intra-modal Attention Mix (IAM) for combining RGB and depth features. The authors claim that IAM consistently outperforms existing fusion methods, including early fusion, late fusion, and attention-based approaches, with gains such as 6.7% average-precision improvement over early fusion on NYUDv2-IS. If the benchmarks are valid, they provide reusable evaluation resources for a field that previously lacked real-world RGB-D instance segmentation datasets. If the module performs as reported, it offers a simple and efficient way to exploit depth for instance segmentation in indoor scenes.","feed_headline":"RGB-D fusion module beats prior methods on three new benchmarks","feed_subtitle":"Three indoor benchmarks and an intra-modal attention mix beat early, late, and attention-based fusion.","key_machinery":"The central mechanism is the Intra-modal Attention Mix (IAM) block, which works on concatenated RGB and depth feature maps written as block matrices $Q = [Q_{rgb}, Q_d]$ and $K = [K_{rgb}, K_d]$. The attention score $QK^T$ then splits exactly into $Q_{rgb}K_{rgb}^T + Q_d K_d^T$, so the model learns spatial relationships inside each modality separately, with no direct cross-modal attention. A mixup-inspired summation blends these intra-modal maps, a channel-wise dynamic fusion module recalibrates the modalities per channel, and the block outputs enhanced RGB features, enhanced depth features, and an aggregated fused feature that feed the detection and segmentation heads of the DETR or SOLQ backbone.","core_discovery":"The paper's central claim is that RGB-D instance segmentation is held back by a lack of real-world instance-level datasets, and that both this gap and the challenge of fusing RGB and depth features can be addressed by two contributions: three new benchmarks (NYUDv2-IS, SUN-RGBD-IS, Box-IS) and the Intra-modal Attention Mix (IAM) module. IAM computes self-attention within each modality by decomposing the query-key product into $Q_{rgb}K_{rgb}^T + Q_dK_d^T$, then recalibrates channel importance with a dynamic fusion step. Across DETR and SOLQ backbones and all three datasets, the paper reports that IAM achieves the highest AP among compared fusion methods, including SA-Gate and CMX, while being computationally lighter because it avoids cross-modal attention terms.","pith_inferences":["Because the NYUDv2-IS and SUN-RGBD-IS masks are derived from connected components of semantic labels, a natural next step is to validate a random subset against human-annotated instances; the paper reports no such verification, and the rankings on these benchmarks could shift if merged or split objects are common.","The same intra-modal decomposition could be applied to other aligned modality pairs such as RGB-Thermal or RGB-surface normals, since the block matrices do not rely on RGB-D-specific assumptions; this is a direct testable extension the paper does not mention.","The baselines SA-Gate and CMX are evaluated with default hyperparameters; a stronger comparison would tune them per dataset, which could reduce the reported margins.","The paper does not report whether the two backbones share weights or train separately; testing weight sharing versus independence would clarify where IAM's benefit comes from."],"forward_implications":["Standardized evaluation becomes possible: future RGB-D instance segmentation work can report numbers on NYUDv2-IS, SUN-RGBD-IS, and Box-IS, which cover real indoor scenes and multiple depth sensors.","IAM can be inserted into existing two-stream detectors as a drop-in block, giving practitioners a cheaper alternative to inter-modal attention that still improves accuracy on small objects and boundaries.","If the gains replicate, depth information becomes more useful for segmentation under low light and texture-ambiguous conditions, where RGB-only features are noisy.","The Box-IS dataset gives the robotics community a dedicated single-class benchmark for box depalletizing, with depth computed from stereo IR images rather than consumer depth post-processing."],"supporting_citations":[{"why":"Supplies the NYUDv2 images and semantic labels from which NYUDv2-IS instance masks are derived.","marker":"[13]"},{"why":"Supplies the SUN-RGBD images and semantic labels from which SUN-RGBD-IS instance masks are derived.","marker":"[14]"},{"why":"Provides the mixup strategy that motivates summing intra-modal attention terms in IAM.","marker":"[15]"},{"why":"Attention-based RGB-D fusion method used as a comparison baseline for IAM.","marker":"[16]"},{"why":"Co-attention RGB-D fusion method used as a comparison baseline for IAM.","marker":"[17]"},{"why":"CMX, a cross-modal fusion method, serves as a state-of-the-art baseline for comparison.","marker":"[3]"},{"why":"SA-Gate, a bi-directional cross-modality fusion method, serves as a state-of-the-art baseline for comparison.","marker":"[4]"},{"why":"DETR is the base architecture that the authors extend with a depth stream and IAM for detection and segmentation.","marker":"[30]"},{"why":"SOLQ is the alternative base architecture, with unified detection and segmentation, also extended with IAM.","marker":"[31]"},{"why":"UniMatch computes high-quality depth maps for Box-IS from stereo IR images.","marker":"[51]"}],"fun_headline_variants":["New RGB-D benchmarks plus IAM fusion top instance segmentation","IAM module and three datasets propel RGB-D instance segmentation","Intra-modal attention beats cross-modal for RGB-D segmentation","Three new RGB-D instance benchmarks and a lighter fusion module"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth instance masks for NYUDv2-IS and SUN-RGBD-IS are generated by extracting connected components from semantic segmentation labels, with no human verification, so touching objects of the same category or occluded objects may be merged or split incorrectly, making the benchmarks unreliable measures of true instance segmentation.","fun_headline_variants_meta":{"raw":{"variants":["New RGB-D benchmarks plus IAM fusion top instance segmentation","IAM module and three datasets propel RGB-D instance segmentation","Intra-modal attention beats cross-modal for RGB-D segmentation","Three new RGB-D instance benchmarks and a lighter fusion module"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00043,"raw_usage":{"total_tokens":2170,"prompt_tokens":892,"completion_tokens":1278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1212}},"tokens_in":508,"tokens_out":1278,"duration_ms":8814,"temperature":1.0,"reasoning_tokens":1212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:39.498137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of images from NYUDv2-IS and SUN-RGBD-IS, have annotators draw instance masks by hand, and compute agreement with the connected-component-derived masks. If a substantial fraction of instances (for example more than 10%) differ in whether they are merged or split, the benchmarks would not measure what they claim, and model rankings on them could be reversed.","supporting_citations":[{"cited_title":"Silberman, D","cited_arxiv_id":null,"evidence_quote":"Supplies the NYUDv2 images and semantic labels from which NYUDv2-IS instance masks are derived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SUN-RGBD images and semantic labels from which SUN-RGBD-IS instance masks are derived."},{"cited_title":"Zhang, M","cited_arxiv_id":null,"evidence_quote":"Provides the mixup strategy that motivates summing intra-modal attention terms in IAM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Attention-based RGB-D fusion method used as a comparison baseline for IAM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Co-attention RGB-D fusion method used as a comparison baseline for IAM."},{"cited_title":"Chen, K.-Y","cited_arxiv_id":null,"evidence_quote":"SA-Gate, a bi-directional cross-modality fusion method, serves as a state-of-the-art baseline for comparison."},{"cited_title":"Carion, F","cited_arxiv_id":null,"evidence_quote":"DETR is the base architecture that the authors extend with a depth stream and IAM for detection and segmentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SOLQ is the alternative base architecture, with unified detection and segmentation, also extended with IAM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UniMatch computes high-quality depth maps for Box-IS from stereo IR images."}],"review_version":1}