{"id":"ca31d9fa-0507-4635-988e-83df5e474b3d","arxiv_id":"2501.06862","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LarvSeg uses image classification data and a category-wise attentive classifier to extend semantic segmentation to 21,000 categories without mask labels for new categories.","lead":"This paper trains image segmentation models using cheap image-level category labels instead of expensive pixel masks, then adds a classifier that focuses supervision on the right image regions. On benchmarks it beats prior open-vocabulary segmenters and shows a model that can segment 21,000 categories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CA-Classifier's central premise—that top-K classification-score pixels localize novel foregrounds—is never validated; since WA847/WA150 original masks exist, a direct localization audit is feasible and would settle whether the reported gains rest on correct supervision.","rationale":"I focused on the CA-Classifier's localization assumption because the paper's distinctive contribution and its headline improvements (6.0 mIoU on A150-novel, 2.1 on A847-novel) are attributed to it. The reader's weakest_assumption identifies the same issue, and I agree. The paper does several things well: the baseline comparison with OpenSeg/ReCo is informative, ablations are consistent, code is released, and the finding that novel-category pixel features are compact is supported by Table 1 and Figure 2. Those supports do not, however, transfer to the memory-bank update rule: §3.2 picks a seed pixel from ground truth, while §3.3 picks top-K pixels from a classifier trained without pixel labels. Because WA150/WA847 are ADE20K/ADEFull with masks available but intentionally not used, the needed validation is cheap and decisive. If the map from top-K pixels to true object regions is poor for small or co-occurring categories, the auxiliary loss in Eq. (5) is applied to the wrong regions, so the reported CA-Classifier gains could be an artifact of the image-level baseline plus regularization rather than of 'category-wise attentive' supervision. This is not an internal inconsistency, but it is an unverified empirical premise at the core of the claim. A localization audit, ideally combined with selecting K and M on a held-out split, would settle it. Therefore I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":10064,"tokens_out":6035,"duration_ms":60290,"concrete_test":"Use the original pixel-level masks of the WA150/WA847 training images (the ADE20K/ADEFull masks that were discarded to create image-level labels). For each novel category, feed the images through the C171 baseline, compute the top-K classification-score pixels specified in §3.3, and measure their IoU/precision against the ground-truth mask, binned by object size and co-occurrence frequency. If mean overlap is low for small or co-occurring categories, the memory bank stores off-target features and the CA-Classifier's attention map in Eq. (4) is supervising wrong regions, directly invalidating the mechanism claimed for the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's distinctive contribution is the category-wise attentive classifier, and its headline gains (6.0 mIoU on A150-novel, 2.1 on A847-novel) are attributed to it. For the CA-Classifier to work as described in §3.3, the memory bank M must store features of the true foreground of each novel category. In training, M is updated with pixel features having the top-K classification scores. This is not the same as the exploratory experiment in §3.2, where a representative pixel is chosen from the ground-truth mask before computing response maps; in the actual algorithm no mask is used. No experiment measures whether top-K-score pixels actually fall inside the novel object. Because WA150/WA847 are formed by discarding the original pixel masks of ADE20K/ADEFull, images contain multiple categories and no explicit negatives: a category such as 'painting' can receive high classification scores on wall or frame pixels, and the memory bank will consolidate those false regions. Equation (4) then subtracts only a single background score map, which cannot suppress all co-occurring categories. If the top-K regions are inaccurate for small objects or tail categories, the auxiliary loss in Eq. (5) reinforces wrong regions, and the Table 4-6 gains may reflect regularization rather than correct category-wise supervision. This is load-bearing because without the CA-Classifier the paper's contribution reduces to a straightforward image-level supervision baseline. The risk is compounded by Table 6, where memory-bank size and top-K are selected using the best novel-category mIoU on the evaluation set itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LarvSeg, a framework for large-vocabulary semantic segmentation that leverages image classification data with image-level labels to augment a segmentation model trained on pixel-level masks. A simple baseline adds an image-level classification loss on globally pooled features. The paper observes that pixel features of categories outside the pixel-label vocabulary are already clustered, and proposes a category-wise attentive classifier (CA-Classifier) that maintains a memory bank of category prototypes, generates attention maps via foreground/background separation, and applies an auxiliary classification loss to attentively pooled features. Experiments on COCO-Stuff with ADE20K/ADEFull (masks discarded) report that LarvSeg improves over the baseline by 6.0 mIoU on A150 novel categories and 2.1 mIoU on A847 novel categories, and the paper claims the first 21K-category semantic segmentation model trained with ImageNet21K, demonstrated qualitatively.","tokens_in":10341,"tokens_out":5825,"duration_ms":58160,"significance":"The paper introduces a promising and practical direction: using abundant image classification data to scale the vocabulary of segmentation models. The baseline itself is simple, strong, and outperforms prior open-vocabulary methods, which is a meaningful result. The CA-Classifier is an intuitive mechanism with consistent but modest gains. However, the evaluation of the CA-Classifier has important gaps: the core localization premise is not directly validated, and key hyperparameters are tuned on the reported test metric. If these issues are addressed, the paper would be a solid contribution. The code release is a strength and supports reproducibility.","major_comments":[{"comment":"The central premise that top-K classification-score pixels localize the true foregrounds of novel categories is not validated. The exploratory experiment in Section 3.2 samples a representative pixel from the ground-truth mask (Eq. 2), whereas the actual training procedure updates the memory bank with pixel features that have the highest classification scores, which may not fall inside the object. Since WA150/WA847 are derived from datasets with full masks, a direct audit is feasible: one could compute the precision or IoU of the top-K regions against the original masks. Without such an audit, the gains attributed to the CA-Classifier could stem from the auxiliary loss acting as a regularizer rather than from correct category-wise supervision, and the causal story of the method is unsubstantiated.","section":"Section 3.3, Eqs. (3)-(5)"},{"comment":"The memory bank size and top-K region area are selected by maximizing the novel-category mIoU on the A150 validation set, which is the same metric used in the headline results (Table 3). This constitutes test-set tuning and can inflate the reported improvement of the CA-Classifier. The paper should either fix these hyperparameters to a priori defaults and report the results, or perform selection on a held-out split and evaluate on another, or report the full sensitivity across a range of values. As written, the 6.0 mIoU improvement on A150 novel categories is not a clean estimate of the method's benefit.","section":"Section 4.5, Table 6"},{"comment":"The auxiliary loss is underspecified. S_cls is not defined as to whether it is the same classification score map used in the baseline or a separate head. The attention map A is defined with a single foreground and background category, but WA150/WA847 are multi-label images containing multiple novel categories; the paper does not explain how fg and bg are determined for each category i, nor how the softmax in Eq. (5) is normalized (over categories or over pixels). The temperature tau is introduced but its value is not listed in Section 4.3. These details are necessary for reproducibility and to assess whether the method is correctly described.","section":"Section 3.3, Eq. (5)"}],"minor_comments":[{"comment":"Equation (5) introduces a temperature tau, but its value is not given in the implementation details; please specify it.","section":"Section 4.3"},{"comment":"The memory bank update rule is not fully described; please state whether features are replaced with moving averages, with what momentum, and whether the top-K selection is performed per image or per batch.","section":"Section 4.3"},{"comment":"The definitions of A150-C171 and A847-C171 are confusing: they say 'validation set with A150-C171 categories' but the column entries are 2K images and 87/809 classes; please clarify that these are the subsets of the validation set whose categories are not in C171.","section":"Table 2"},{"comment":"The description 'we set other categories in the image as ignore labels' is ambiguous for multi-label classification; please state explicitly whether the loss is binary cross-entropy per category with ignore masks.","section":"Section 3.1"},{"comment":"ADEFull is cited as [29,6], but reference [6] (Cheng et al., 'Per-pixel classification is not all you need') does not introduce ADEFull; please cite the appropriate ADE20K/Scene Parsing dataset papers.","section":"References"},{"comment":"The claim of being the 'first 21K-category semantic segmentation model' is supported only by qualitative examples; consider softening the claim or providing a quantitative evaluation on a subset with available masks (e.g., ImageNet validation with object masks).","section":"Abstract / Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is defensible but the evaluation has two load-bearing issues: the CA-Classifier's localization assumption is not audited despite the data being available, and the main hyperparameters are tuned on the reported test metric. These should be addressed before publication. The baseline result alone is interesting, but the paper's novelty rests on the CA-Classifier, so the gaps are important. The 'first 21K model' claim may also need a prior-art check from the editor's side."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a referee's time. The paper's core claim—that image classification data can push segmentation vocabularies to 21K categories—is supported by consistent experiments, and the baseline alone beats open-vocabulary methods by a large margin. The CA-Classifier is a new mechanism, and its gains (6.0 mIoU on A150 novel, 2.1 on A847 novel) are consistent across settings. The authors also ship a promised code release and provide a reasonable exploratory experiment showing that novel-category pixels cluster in feature space. That is real work, and the first 21K-category segmenter is a legitimate milestone.\n\nThe soft spots are what you'd expect from the stress-test note. The CA-Classifier's memory bank is updated using pixels with the top-K classification scores, but the paper never verifies that those pixels actually fall on the novel object. Section 3.2's exploratory experiment uses ground-truth masks to pick the representative pixel; the actual training does not. Because WA150/WA847 are formed by discarding existing pixel masks, a direct localization audit is easy to run, and it would settle whether the auxiliary loss supervises correct regions or just acts as a regularizer. This is a load-bearing gap: if the top-K pixels are wrong for small or co-occurring categories, the reported gains may not reflect category-wise supervision at all.\n\nTwo more concerns, in proportion: Table 6 selects memory-bank size and top-K using novel-category mIoU on the evaluation set itself, which inflates the headline numbers; and the 21K claim is only qualitative, so we don't know how well the model actually handles that vocabulary. Error bars are absent, though that is common in this subfield.\n\nNone of this sinks the paper. The baseline result and the overall direction are solid, and the CA-Classifier is a plausible mechanism that deserves a proper test. The fix is concrete: run the localization audit, pick hyperparameters on a held-out split, and report at least a small quantitative 21K benchmark. This paper is for people working on large-vocabulary or weakly supervised segmentation, and I'd send it to peer review with those requests.","headline":"A solid, useful extension of Detic-style image-level supervision to segmentation, with a genuinely new CA-Classifier and a first 21K-category segmenter; the headline gains are real but the mechanism is under-validated and hyperparameters are tuned on the eval set.","tokens_in":659,"tokens_out":1811,"would_cite":false,"duration_ms":34246,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Image-level category labels can scale semantic segmentation to 21K classes.","keywords":["Large vocabulary semantic segmentation","Category-wise attentive classifier","Image-level supervision","Semantic segmentation","ImageNet21K","Weakly supervised segmentation","Memory bank prototypes"],"falsifier":"Evaluate LarvSeg's novel-category mIoU separately for small objects and for categories that usually co-occur with larger background classes, such as 'painting' on a 'wall' or 'mug' on a table. If the CA-Classifier's gain over the simple baseline reverses or disappears on that subset, the assumption that top-$K$ score pixels mark foreground is falsified; the baseline could still work, but the attentive classifier would not be adding correct localization.","tokens_in":9841,"feed_emoji":"🖼️","tokens_out":10574,"duration_ms":91760,"temperature":0.7,"pith_summary":"The paper tries to establish that image classification data -- images labeled with category tags rather than pixel masks -- can be used to scale semantic segmentation to vocabularies far beyond the few hundred classes that have mask annotations. It proposes LarvSeg, which trains a segmentation model jointly on pixel-level segmentation data and image-level classification data, and reports that this simple baseline already beats language-guided open-vocabulary segmentation models on categories never seen with masks. The paper further claims that a model trained on only base categories already groups pixels of unseen categories, so a category-wise attentive classifier built from memory-bank prototypes can apply classification supervision to the correct regions. On the reported benchmarks this adds 6.0 mIoU over the baseline on A150 novel categories and 2.1 mIoU on A847 novel categories, and with ImageNet21K it yields, for the first time, a semantic segmentation model for 21K categories. The central thesis is that cheap image-level labels are a practical route to large-vocabulary segmentation.","feed_headline":"Image tags alone teach a model to segment 21K classes","feed_subtitle":"A category-wise attentive classifier targets labels to the right pixels, lifting novel-category mIoU by up to 6.0","key_machinery":"The central mechanism is the category-wise attentive classifier (CA-Classifier), a memory-bank prototype module that turns image-level labels into pixel-level supervision. For each novel category it stores representative features taken from pixels with the top-$K$ classification scores across images, computes for each pixel the average cosine similarity to those prototypes, forms a category-wise attention map $A = \\mathrm{sigmoid}(\\mathrm{norm}(S^{fg}_{\\mathrm{mem}} - S^{bg}_{\\mathrm{mem}}))$ that strengthens foreground and suppresses background, and applies an auxiliary image-level classification loss to the attentively pooled score map. This is what lets the model supervise the precise regions of categories that never had mask labels.","core_discovery":"The paper's central claim is that pixel-level supervision is not necessary for expanding a segmentation model's vocabulary: image-level labels suffice, provided the supervision is aimed at the right pixels. The authors show that a segmenter trained on 171 fully labeled classes, together with weakly labeled classification images, can segment novel categories, and they observe that feature pixels of novel categories are already grouped even when the model never saw those categories. Building on that observation, the category-wise attentive classifier maintains per-category prototypes in a memory bank and uses a foreground-strengthening, background-suppressing attention map so that each novel category's auxiliary image-level loss is concentrated on its actual region. The paper reports that this method outperforms the simple baseline on novel categories, and that training with ImageNet21K yields a 21K-category semantic segmentation model, which the paper states is the first of its kind.","pith_inferences":["A reader could take the paper one step further: since the baseline already beats text-supervised models without any language supervision, the practical bottleneck for large-vocabulary segmentation may be cheap category-level labels rather than visual-language alignment; the paper does not state this conclusion explicitly.","A testable extension is to shrink the classification dataset per category: if prototypes built from only a few images still lift novel-category mIoU, the approach becomes usable where large classification corpora are unavailable.","The evidence that pixel features of unseen categories cluster suggests the same memory-bank mechanism might transfer to panoptic or instance segmentation, where category prototypes could seed region proposals; the paper does not explore that.","A stress test on small or heavily co-occurring novel objects would bound the CA-Classifier's added value: a failure there would not refute the baseline but would show the attentive module's localization assumptions."],"forward_implications":["Segmentation vocabularies can be expanded with image classification datasets that already exist, avoiding the cost of new pixel-level mask annotation.","Training with segmentation masks plus image-level labels outperforms language-guided open-vocabulary models on unseen categories, suggesting image-level supervision is a stronger recognition signal than text alignment.","Balanced, object-centric classification data such as ImageNet21K improves novel-category accuracy more than model architecture changes alone, pointing to data quality as the key lever.","The category-wise attentive classifier improves segmentation on both multi-label and single-label classification data, and the two kinds of data combine for further gains.","A 21K-category segmenter can recognize fine-grained classes such as 'polyphemus moth,' showing the scaling path extends beyond coarse categories."],"supporting_citations":[{"why":"Establishes that image-level classification data can scale object detection to 21K categories, the recipe LarvSeg adapts to segmentation.","marker":"[31]"},{"why":"Provides ImageNet21K, the 21,841-category image-level dataset used for the 21K segmentation model and for pretraining the backbone.","marker":"[7]"},{"why":"Supplies COCO-Stuff (C171), the pixel-level segmentation data used for base supervision and as evaluation context.","marker":"[3]"},{"why":"Supplies ADE20K and ADEFull (A150/A847), the evaluation benchmarks and weak image-level labels WA150/WA847.","marker":"[29]"},{"why":"Provides the language-guided DenseCLIP baseline and the comparison used in the exploratory pixel-grouping experiment.","marker":"[30]"},{"why":"Provides LSeg, a language-driven segmentation baseline the paper reproduces and compares against.","marker":"[14]"},{"why":"Provides ZSBaseline, a zero-shot segmentation baseline the paper reproduces with ViT-B/16 for fair comparison.","marker":"[27]"},{"why":"Provides OpenSeg, the state-of-the-art open-vocabulary baseline that LarvSeg outperforms despite OpenSeg using far more web-scale paired data.","marker":"[10]"}],"fun_headline_variants":["Image tags alone teach segmentation of 21K classes","Category-wise attention turns image tags into pixel labels","LarvSeg: 21K-class segmentation from image-level labels","No masks needed: image tags scale segmentation vocabulary","Weak labels, strong segmentation: 21K categories via attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the pixels with the highest classification scores for a category are actually the category's foreground pixels, so the memory-bank prototypes and attention map apply supervision to the right regions rather than to background or frequently co-occurring objects.","fun_headline_variants_meta":{"raw":{"variants":["Image tags alone teach segmentation of 21K classes","Category-wise attention turns image tags into pixel labels","LarvSeg: 21K-class segmentation from image-level labels","No masks needed: image tags scale segmentation vocabulary","Weak labels, strong segmentation: 21K categories via attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1405,"prompt_tokens":1003,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":619,"tokens_out":402,"duration_ms":4028,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:28.326907+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate LarvSeg's novel-category mIoU separately for small objects and for categories that usually co-occur with larger background classes, such as 'painting' on a 'wall' or 'mug' on a table. If the CA-Classifier's gain over the simple baseline reverses or disappears on that subset, the assumption that top-$K$ score pixels mark foreground is falsified; the baseline could still work, but the attentive classifier would not be adding correct localization.","supporting_citations":[{"cited_title":"In: European Conference on Com- puter Vision","cited_arxiv_id":null,"evidence_quote":"Establishes that image-level classification data can scale object detection to 21K categories, the recipe LarvSeg adapts to segmentation."},{"cited_title":"In: Proceedings of the IEEE conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Supplies COCO-Stuff (C171), the pixel-level segmentation data used for base supervision and as evaluation context."},{"cited_title":"In: Proceedings of the IEEE conference on computer vision and pattern recognition (2021)","cited_arxiv_id":null,"evidence_quote":"Provides the language-guided DenseCLIP baseline and the comparison used in the exploratory pixel-grouping experiment."}],"review_version":1}