{"id":"539e6e9b-d56c-4865-acc0-e1f5229707a1","arxiv_id":"2411.17536","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-task learning method that exchanges weak supervision between bounding boxes and segmentation masks improves detection and segmentation on partially annotated data.","lead":"This paper trains object detection and semantic segmentation together when each training image is labeled for just one of the two tasks. It adds weak losses that let boxes help train masks and masks help train boxes, and the combined method improves both tasks on the COCO benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"COCO gains may be confounded: BoMBo adds an attention module absent from the MTL baseline, so the reported improvements may not isolate the proposed weak losses.","rationale":"The reader's weakest_assumption correctly identifies a stated limitation (shared class space and domain), but it is not the most load-bearing concern for the central claim. The central claim is that the proposed weak losses improve multi-task partially supervised learning. The principal evidence (Table 5 on COCO) is presented as a comparison between BoMBo and an MTL baseline. However, BoMBo includes a modification to the segmentation head (the attention module) that the baseline lacks. This architectural difference is explicit in Section 6. Because the attention module is trained with a box-derived loss (Lα) as part of Box-for-Mask, it is intertwined with the weak-loss intervention. A rigorous evaluation must separate the effect of the extra model capacity/architecture from the effect of the cross-task supervision. The paper's ablations in Tables 1 and 4 are informative, but Table 1 concerns only detection (no attention module), and Table 4 compares against baselines that may also lack the attention module, leaving the same confound. Thus, the COCO numbers do not conclusively support the claim that the weak losses themselves are responsible for the gains. A deconfounded ablation would settle this. The paper is otherwise competent, with useful ablations and qualitative analyses, so keeping the verdict at CONDITIONAL (pending the deconfounding experiment) is appropriate rather than outright rejection.","tokens_in":12346,"tokens_out":7196,"duration_ms":68516,"concrete_test":"Run the COCO experiment with a fourth condition: MTL baseline plus the attention module, where the attention map (Eq. 3) is trained only on ground-truth semantic masks (or set to identity), so no box-derived pseudo-masks are used for Lα and no cross-task weak losses are applied. Compare its detection mAP and segmentation IoU to the reported MTL and BoMBo rows in Table 5. Also, run Mask-for-Box alone (detection side) without the attention module to verify that the detection gains persist. If the attention-module-only row matches BoMBo's segmentation, the weak losses are not the driver of the observed gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 5 reports BoMBo gains over an MTL baseline on COCO, but the comparison is confounded by an architectural change. Section 6 states that the network is 'kept unchanged from [16] except an additional attention module in the segmentation head.' The MTL baseline (from [16]) therefore lacks the attention module that BoMBo adds as part of Box-for-Mask. The COCO segmentation improvements (54.535 to 58.466 for ResNet50) could stem from this extra module alone, rather than from the proposed weak losses. The paper does not include a control row: an MTL baseline with the attention module but without cross-task pseudo-targets. Without such an ablation, the core attribution of Table 5 to the weak losses is not established. The class-space assumption (the reader's weakest_assumption) is real but secondary: it affects generality, not the internal validity of the COCO comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies multi-task partially supervised learning (MTPSL) for object detection and semantic segmentation, where each training image is annotated for only one of the two tasks. It proposes two modules: Mask-for-Box (M4B), which refines circumscribed rectangles of ground-truth mask connected components using predicted boxes to generate localization-only pseudo-labels for the detector, and Box-for-Mask (B4M), which builds pseudo-masks from ground-truth boxes (box-filling plus GrabCut) and trains the segmentation head with a cross-entropy loss on the coarse pseudo-mask, an attention loss on box-shaped masks, and a triplet loss on embeddings. The two modules are combined into BoMBo and evaluated on VOC and COCO with ResNet and Swin backbones. The reported COCO results show consistent gains over the multi-task baseline, while VOC results are mixed.","tokens_in":74,"tokens_out":6092,"duration_ms":120123,"significance":"If the reported gains are robust, the work is a useful contribution to exploiting heterogeneous partially annotated datasets in multi-task learning, building on prior MTPSL work. The paper contains several strengths: design-level ablations for each module (Tables 1, 3, 4), a TIDE error analysis for the detection component, experiments across multiple backbones, and a commitment to release source code and data splits. However, the main COCO comparison is confounded by an architectural change (an added attention module) that is not present in the baseline, and the VOC results are not consistently favorable, so the evidence for the central claim is currently partial.","major_comments":[{"comment":"The comparison between BoMBo and the MTL baseline in Table 5 is confounded by an architectural change. Section 6 states that the network is 'kept unchanged from [16] except an additional attention module in the segmentation head.' The MTL baseline (from [16]) therefore lacks this attention module. The reported COCO segmentation improvements (e.g., ResNet50 from 54.535 to 58.466) could stem from the additional module alone, independently of the proposed cross-task losses. A necessary control experiment is an MTL baseline with the attention module added but without any Box-for-Mask or Mask-for-Box objectives. Without this control, the attribution of the COCO gains to the weak losses is not established. The same confound affects Table 4, where the '+Lα' row adds both the attention module and the attention loss relative to the baseline, so the effect of Lα is not isolated.","section":"Section 6, Table 5"},{"comment":"The VOC results do not support the abstract's implication of generally favorable performance. On VOC, BoMBo improves both tasks only for SwinL; on ResNet50 both detection mAP and segmentation IoU decrease (55.174 to 54.885 and 75.658 to 74.861), and on SwinT/SwinB the segmentation IoU decreases despite detection gains. The paper notes an 'imbalance problem' in passing, but does not analyze why the combination helps on COCO and hurts on VOC, or why the effect is backbone-dependent. Since the central claim is that cross-task weak losses improve MTPSL, the VOC results are a load-bearing inconsistency that needs either a technical explanation (e.g., dataset-domain shift, class-space mismatch) or a tempered statement of the claim.","section":"Table 5"},{"comment":"All experimental numbers are reported from a single run with no error bars or multiple seeds. This is particularly concerning for the small differences in Table 5, such as VOC SwinB detection (58.267 vs 58.259) and the mixed VOC segmentation changes. Without variance estimates, the reader cannot assess whether any of the gains or losses are statistically meaningful. At least the main BoMBo versus MTL comparisons should be repeated with multiple seeds and the mean and standard deviation reported.","section":"Section 4.1"}],"minor_comments":[{"comment":"There is a typo in the abstract: 'Bo x-for-Mask' should read 'Box-for-Mask'.","section":"Abstract"},{"comment":"The summation notation 'k∈0...nB' is ambiguous; it should be 'k = 0, ..., nB−1'.","section":"Eq. (4)"},{"comment":"The paper does not specify how the triplet-loss keys are stored and updated across batches, beyond stating that the EMA network is used. Clarify whether the keys are embeddings from the EMA network on previous-batch images and how the key bank is maintained.","section":"Section 3.3"},{"comment":"The thresholds in the Mask-for-Box algorithm (IoU 0.6/0.8, confidence 0.1/0.4/0.5, NMS 0.4) are introduced without justification or sensitivity analysis. A small study varying these parameters would improve confidence in the method's robustness.","section":"Algorithm 1"},{"comment":"The informal phrase 'a predicted box should have at least 2 sides touching those of a reference box' is only made precise in Algorithm 1's touch function. The main text should refer explicitly to the algorithm's definition to help the reader.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"This is a method paper with a plausible idea and a solid set of ablations, but the key COCO comparison is currently confounded by the addition of an attention module. The missing control experiment (MTL plus attention without weak losses) is essential and should be requested. The VOC results also need to be addressed analytically rather than dismissed as an imbalance. The paper is within scope for a computer vision journal, but needs these revisions before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful paper for the multi-task partially supervised learning crowd, not a breakthrough. Mask-for-Box is the genuinely new piece: it takes connected-component boxes from ground truth masks and splits, merges, or filters them using the detector's own predictions, then trains only localization on the refined boxes. Box-for-Mask is a competent assembly of existing box-driven segmentation losses (Box2Seg-style attention, filling-rate ideas, triplet loss) adapted to the partial-label setting. The per-module ablations in Tables 1-4 are informative and mostly support the design choices. I also credit the authors for reporting the mixed VOC numbers and for shipping code and data splits.\n\nThe soft spot is the one the stress-test flags, and it lands. In Table 5, BoMBo is compared against the [16] MTL baseline, but Section 6 says the network is unchanged from [16] except an additional attention module in the segmentation head. That attention module is part of Box-for-Mask. So the COCO gains (e.g., ResNet50 segmentation IoU from 54.5 to 58.5) are not cleanly attributable to the weak losses; an attention-only control row, MTL plus attention without cross-task pseudo-targets, is missing. This is fixable but load-bearing: it weakens the central attribution claim as stated. The class-space/domain assumption is acknowledged in the Discussion and is more a generality limit than an internal flaw.\n\nOther concerns are minor to moderate: no error bars or multiple seeds anywhere, a stack of hand-set thresholds (IoU 0.6/0.8, confidence 0.1/0.4/0.5, NMS 0.4) with no sensitivity analysis, and VOC results are mixed except on SwinL, where BoMBo helps both tasks. None of these kill the paper; they mean the evidence is suggestive rather than definitive.\n\nWho is this for: researchers working on weakly supervised or partially supervised detection/segmentation, especially those building on MTPSL baselines. It deserves a serious referee. My recommendation: send it out, but insist on an attention-module control ablation and variance reporting before acceptance. The mixed VOC numbers alone should not sink it; the confounded comparison needs to be fixed.","headline":"Useful MTPSL paper with a genuinely new Mask-for-Box refinement, but the headline COCO comparison is confounded by an un-ablated attention module added to the segmentation head.","tokens_in":5,"tokens_out":2136,"would_cite":true,"duration_ms":51103,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Weak label transfer between detection and segmentation improves COCO multi-task learning","keywords":["multi-task learning","partially supervised learning","object detection","semantic segmentation","weak supervision","pseudo-labels","Box-for-Mask","Mask-for-Box"],"falsifier":"Run BoMBo on two datasets with disjoint or only partially overlapping class sets, for example COCO detection with Cityscapes segmentation, keeping everything else fixed; if the gains on COCO disappear or turn negative, the shared-class-space assumption is what carries the result. Alternatively, replace the refined pseudo-boxes with the unrefined circumscribed rectangles of the ground-truth masks and check whether the detection margin on COCO collapses, isolating the contribution of the refining step.","tokens_in":12111,"feed_emoji":"🎯","tokens_out":6590,"duration_ms":55552,"temperature":0.7,"pith_summary":"The paper studies multi-task partially supervised learning, where each training image is annotated for exactly one task, either object detection or semantic segmentation, and asks whether the annotations of one task can be used to train the other task's head. It proposes two modules: Mask-for-Box, which refines the circumscribed rectangles of a segmentation mask's connected components using the model's own predicted boxes and trains detection with the localization loss only, and Box-for-Mask, which builds pseudo-masks from ground-truth boxes, including a box-filled mask for attention, a coarse mask for cross-entropy, and a triplet loss on embeddings, to train segmentation. Combined as BoMBo, the method improves both tasks over the multi-task baseline on COCO across all tested backbones, with detection mAP rising from 17.2 to 19.1 and segmentation IoU from 54.5 to 58.5 on ResNet50. On VOC the gain is mixed, improving both tasks only for the largest backbone. The paper's point is that noisy but weakly relevant targets from the other task's ground truth are usable if the noise is filtered by the network's own predictions and by attention modulation.","feed_headline":"Weak label swap lifts COCO detection and segmentation","feed_subtitle":"Using one task's ground truth to train the other's head improves multi-task partially supervised learning on COCO.","key_machinery":"The load-bearing mechanism is a pair of label-refinement modules that convert one task's ground truth into targets for the other task's head. Mask-for-Box uses a matching and refining algorithm: reference boxes are extracted as circumscribed rectangles of connected components in the ground-truth mask; predicted boxes with confidence above thresholds are matched to references, used to split multi-instance references, merge fragmented ones into a smallest covering box, or kept as is, and the surviving refined boxes are filtered by non-maximum suppression; only the localization loss is applied. Box-for-Mask generates two pseudo-masks from each ground-truth box, a box-shaped mask of filled categories that trains an auxiliary attention map via mean-squared error, and a coarse mask restricted to the box interior that trains the main cross-entropy loss, plus a triplet loss that anchors each box's mean embedding of correctly predicted pixels to a same-class key and pushes it away from a different-class key. The attention-modulated prediction $M = M_l \\otimes \\alpha$ is what lets the noisy pseudo-masks improve segmentation instead of degrading it.","core_discovery":"On the paper's own terms, the central discovery is that cross-task weak supervision works in multi-task partially supervised learning if the transfer is designed around what each annotation type can and cannot say. A ground-truth mask gives correct categories and an object's extent but not instance boundaries, so Mask-for-Box takes the circumscribed rectangles of connected components, which may be fragmented or contain many instances, and re-localizes them using the network's predicted boxes, splitting, merging, or keeping boxes according to overlap and a two-sided touch criterion, then supervises the detection head with only the localization loss, because adding classification loss hurts. A ground-truth box gives the object's category and an upper bound on its extent but no pixel-level boundary, so Box-for-Mask generates a box-shaped mask used to train an attention map, a coarse pseudo-mask used for cross-entropy, and a triplet loss pulling embeddings of pixels inside a box toward same-class embeddings and away from other classes; the segmentation prediction is the logits modulated by the learned attention. Combining the two losses with lambda equal to 2, BoMBo outperforms the multi-task baseline on COCO for detection and segmentation on every backbone tested, while on VOC it improves both tasks only for Swin-L and is mixed otherwise.","pith_inferences":["If the shared-class-space assumption holds, the same recipe could be tried for other task pairs with complementary annotation granularity, such as instance segmentation and depth, where one task's dense ground truth could be converted into box-like or mask-like targets for the other.","The localization-only finding hints that pseudo-targets should carry only the information the target task lacks, not the full supervision; a testable extension would be to decay the classification weight on pseudo-boxes gradually as the network's own classifiers mature.","The mixed VOC results suggest a concrete diagnostic: measure the per-class overlap between the detection and segmentation label distributions, and use that overlap to gate whether BoMBo modules are switched on for a given class.","Because Mask-for-Box improves most on miss and false-negative errors, it could be attached to a detector that is already strong at classification but weak at recall, using segmentation data to grow recall without touching classifier training."],"forward_implications":["In partially annotated settings where detection and segmentation datasets share a class space, one task's annotations can genuinely expand the other's training data, improving both tasks on COCO at one-eighth data scale.","Classification supervision on pseudo-boxes is counterproductive; the useful signal from mask-derived boxes is their localization, which should be trained with the localization loss only.","Box-derived pseudo-masks do not help when they directly drive the segmentation cross-entropy; they work only when mediated by an attention map trained on the box-shaped filling.","The gains transfer across backbones on COCO, from ResNet18 and ResNet50 to Swin-T, Swin-B, and Swin-L, suggesting the mechanism is not tied to one architecture.","Combining both modules requires balancing the tasks, with lambda set to 2, and the mixed VOC results indicate the balance interacts with dataset difficulty or domain closeness."],"supporting_citations":[{"why":"Establishes the multi-task partially supervised learning setup for detection and segmentation, supplies the baseline architecture and data splits.","marker":"[16]"},{"why":"Introduces the multi-task partially supervised learning paradigm and the notion of partially annotated data.","marker":"[19]"},{"why":"GrabCut is used to generate coarse pseudo-masks in the Box-for-Mask module.","marker":"[31]"},{"why":"Box2Seg provides the attention-weighted loss and box-shaped mask idea that the Box-for-Mask attention module builds on and compares against.","marker":"[15]"},{"why":"Unbiased Teacher supplies the EMA and pre-trained teacher knowledge-distillation baselines against which Mask-for-Box refinement is tested.","marker":"[25]"},{"why":"Microsoft COCO is the dataset on which the final BoMBo comparison is reported.","marker":"[20]"},{"why":"Pascal VOC is the dataset for the main experiments with the detection and segmentation split.","marker":"[7]"},{"why":"SBD provides the extra semantic segmentation ground truth used to build the VOC partially annotated training splits.","marker":"[10]"}],"fun_headline_variants":["Weak label swap lifts COCO detection and segmentation","BoMBo: weak supervision from boxes to masks and back","Cross-task weak labels boost COCO multi-task learning","Mask-to-box and box-to-mask weak losses improve COCO","Swap task annotations to train the other head on COCO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the detection and segmentation datasets share the same semantic class space and come from the same image domain, because the pseudo-targets inherit their category labels from the other task's ground truth; if the class sets differ, the weak losses would train wrong categories.","fun_headline_variants_meta":{"raw":{"variants":["Weak label swap lifts COCO detection and segmentation","BoMBo: weak supervision from boxes to masks and back","Cross-task weak labels boost COCO multi-task learning","Mask-to-box and box-to-mask weak losses improve COCO","Swap task annotations to train the other head on COCO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":3297,"prompt_tokens":993,"completion_tokens":2304,"prompt_tokens_details":{"cached_tokens":896},"prompt_cache_hit_tokens":896,"prompt_cache_miss_tokens":97,"completion_tokens_details":{"reasoning_tokens":2223}},"tokens_in":97,"tokens_out":2304,"duration_ms":20941,"temperature":1.0,"reasoning_tokens":2223,"cache_read_input_tokens":896,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:01:12.701258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BoMBo on two datasets with disjoint or only partially overlapping class sets, for example COCO detection with Cityscapes segmentation, keeping everything else fixed; if the gains on COCO disappear or turn negative, the shared-class-space assumption is what carries the result. Alternatively, replace the refined pseudo-boxes with the unrefined circumscribed rectangles of the ground-truth masks and check whether the detection margin on COCO collapses, isolating the contribution of the refining step.","supporting_citations":[{"cited_title":"Data exploitation: multi-task learning of object detection and semantic segmentation on partially annotated data","cited_arxiv_id":null,"evidence_quote":"Establishes the multi-task partially supervised learning setup for detection and segmentation, supplies the baseline architecture and data splits."},{"cited_title":"Learning Multiple Dense Prediction Tasks from Partially Annotated Data","cited_arxiv_id":null,"evidence_quote":"Introduces the multi-task partially supervised learning paradigm and the notion of partially annotated data."},{"cited_title":"GrabCut: Interactive Fore- ground Extraction Using Iterated Graph Cuts","cited_arxiv_id":null,"evidence_quote":"GrabCut is used to generate coarse pseudo-masks in the Box-for-Mask module."},{"cited_title":"Box2Seg: Attention Weighted Loss and Discriminative Feature Learning for Weakly Supervised Segmentation","cited_arxiv_id":null,"evidence_quote":"Box2Seg provides the attention-weighted loss and box-shaped mask idea that the Box-for-Mask attention module builds on and compares against."},{"cited_title":"Unbiased Teacher for Semi-Supervised Object Detection","cited_arxiv_id":null,"evidence_quote":"Unbiased Teacher supplies the EMA and pre-trained teacher knowledge-distillation baselines against which Mask-for-Box refinement is tested."},{"cited_title":"Microsoft COCO: Common Objects in Context","cited_arxiv_id":null,"evidence_quote":"Microsoft COCO is the dataset on which the final BoMBo comparison is reported."},{"cited_title":"The Pascal Visual Object Classes (VOC) Challenge","cited_arxiv_id":null,"evidence_quote":"Pascal VOC is the dataset for the main experiments with the detection and segmentation split."},{"cited_title":"Semantic contours from inverse detectors","cited_arxiv_id":null,"evidence_quote":"SBD provides the extra semantic segmentation ground truth used to build the VOC partially annotated training splits."}],"review_version":1}