{"id":"be2b6443-98d7-4a3b-a4c5-133a7fb651ac","arxiv_id":"2501.09688","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PartCATSeg improves open-vocabulary part segmentation by separating object- and part-level cost volumes, adding a compositional loss, and injecting DINO structural guidance, achieving over 10% h-IoU gains on three benchmarks.","lead":"This paper presents PartCATSeg, a method for recognizing and segmenting object parts in images of categories never seen during training. It combines cost aggregation, a compositional loss, and DINO features to improve open-vocabulary part segmentation, reporting large gains over prior methods on three benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pred-All gains on ADE20K-Part-234 hinge on the compositional mapping and are not tested under partial part annotation; a masked-loss ablation would settle whether Eq. 13 distorts or sharpens part confidence.","rationale":"The reader identified the compositional mapping and partial-part annotation as the weakest assumption; I agree. The paper's strongest claim is the new SOTA baseline with large h-IoU gains, and those gains are claimed to come from the compositional loss and disentangled costs. The load-bearing risk is that Eq. 13 is a hard per-pixel equality constraint between the object distribution and the summed part distribution. In OVPS benchmarks, ground-truth part masks are incomplete: not every part of every object is annotated, and common parts (e.g., wheels) may be occluded or absent. Under such partial annotation, the loss cannot distinguish 'this pixel is part p' from 'this pixel is part of object o but its specific part label is missing,' so it can inflate arbitrary part probabilities for unannotated object pixels. The provided ablations do not test this: they toggle the loss but do not mask it per-pixel and do not report coverage statistics. Additionally, the Pred-All comparison on ADE20K has only one baseline (PartCLIPSeg) with much lower seen mIoU, so the headline 12.81-point gain could partly reflect protocol/baseline incompleteness rather than the proposed mechanism. This does not require rejecting the paper; it requires a conditional accept with a targeted experiment. The reader's CONDITIONAL verdict remains appropriate, with the concrete masked-loss ablation as the decisive check.","tokens_in":24361,"tokens_out":1944,"duration_ms":19235,"concrete_test":"Run Pascal-Part-116 training with Lcomp applied only at pixels whose ground-truth part belongs to the annotated object-part vocabulary (masked Lcomp), and compare against the current global Lcomp and against w/o Lcomp in Table 5. If masked Lcomp preserves or improves h-IoU, the Eq. 13 prior was being distorted by partial annotation. Independently, recompute ADE20K Pred-All with PartGLEE or another Pred-All baseline, or add a leak check: evaluate PartCATSeg with the object cost volume removed and report whether the 12.81-point gain persists.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is state-of-the-art OVPS, supported mainly by large Pred-All h-IoU gains (Tables 1-3). The most load-bearing assumption is Eq. 13: a deterministic mapping from each object-specific part class to its parent object, combined with a per-pixel Jensen-Shannon loss that forces the object softmax over CObj to match the summed part softmax over CObj-Part. This prior only holds if every pixel has exactly one true parent object and if the part set exhaustively covers the object. In these benchmarks both conditions are approximate. Pascal-Part-116 does not annotate every part in every image (e.g., a side-view aeroplane may show no engine), and the paper's own qualitative examples show regions the model labels 'unsegmented.' If a pixel's ground-truth part is absent from the training mask vocabulary, the loss gradient pushes PObj-Part to match PObj exactly, which may suppress the 'background'/'none' mass and encourage confident mislabeling of unannotated regions as a seen part. The ablation (Table 5) only toggles the loss on/off and compares L1 vs softmax normalization; it does not vary the mapping, mask out unannotated pixels, or analyze per-image part-coverage rates. A second correlated risk is benchmark completeness: PartCLIPSeg is the only Pred-All baseline on ADE20K, and its low seen mIoU (14.15) suggests the protocol itself may penalize joint object+part models differently. These two concerns are separable: the first is an internal failure mode of Eq. 13, the second is an external comparison gap. The internal one is more load-bearing because it could invalidate the claimed mechanism even if the benchmark numbers are reproduced.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PartCATSeg extends CAT-Seg's cost-aggregation machinery to open-vocabulary part segmentation. The method computes separate object-level and part-level image-text cost volumes (Sec. 3.3), fuses them into an object-specific part cost volume (Sec. 3.4), injects DINOv2 features as structural guidance into spatial aggregation (Sec. 3.6), and trains with an auxiliary compositional loss that forces the softmax distribution over object classes to equal the part-class distribution aggregated through a fixed part-to-object mapping M (Sec. 3.5, Eqs. 13-14). Experiments on Pascal-Part-116, ADE20K-Part-234, and PartImageNet report large gains over prior methods in both Pred-All and Oracle-Obj settings, especially on unseen classes, with component-level ablations for the compositional loss and structural guidance.","tokens_in":24853,"tokens_out":7192,"duration_ms":65492,"significance":"If the reported numbers are accurate, PartCATSeg is a clear advance for zero-shot part segmentation: the Pred-All h-IoU gains of 15.1, 12.8, and 27.8 points over the strongest available baseline on the three main benchmarks are large and qualitatively visible in the supplied figures. The strengths are the clear architectural decomposition, the release of code, the honest limitations section, and the component-level ablations that separately attribute gains to cost aggregation, DINO guidance, and the compositional loss. The main novelty is moderate--the framework is a direct extension of the authors' earlier PartCLIPSeg and borrows CAT-Seg's cost aggregation--but the empirical margin is substantial. The central risk is that the largest gains come from the compositional loss, whose premise (complete, one-to-one part-to-object coverage at every pixel) is only approximately true in the benchmarks; this needs targeted experiments before the specific mechanism can be considered established.","major_comments":[{"comment":"The compositional loss is applied at every spatial location and equates PObj(i) with the sum of part probabilities over M^{-1}(o). The paper never states whether CObj includes a background/void class. If it does not, background pixels have an arbitrary object-class distribution, and Lcomp will push part probabilities to match it, encouraging hallucinated parts. If it does, Eq. (13) assigns zero aggregated mass to the background class because no part maps to it, while PObj may carry large background mass, so the equality cannot hold except by suppressing the background. Moreover, the benchmarks do not annotate all parts in every image (e.g., a side-view aeroplane on Pascal-Part-116 can have no visible engine), so the premise 'parts collectively compose the object' is violated for many pixels. The paper should specify the class set, mask the loss to pixels that have a valid part annotation (or to a foreground object mask), and report the fraction of pixels affected. Without this, the gradient from Lcomp may distort part confidences on unseen classes rather than sharpen them.","section":"Sec. 3.5, Eqs. (11)-(14)"},{"comment":"The ablation of Lcomp toggles the loss on/off and compares softmax vs L1 normalization, but it does not test the sensitivity of Eq. (13) to the fixed mapping M or to incomplete part annotations. The mapping M is described as 'predefined' and is never justified or varied; part classes that are shared across multiple object classes or part sets that only partially cover an object would change the aggregated target. Since the largest reported gains are on unseen classes (Tables 1-3), it is important to ablate (a) a masked version of Lcomp that ignores unannotated pixels, (b) a random or perturbed mapping M as a negative control, and (c) per-image part-coverage statistics. This would determine whether the loss sharpens genuine part-object composition or simply suppresses low-confidence predictions.","section":"Sec. 4.3, Table 5"},{"comment":"In the structural guidance ablation, applying DINO guidance to both TSA_Obj and TSA_Part (the full model) yields seen mIoU 52.62, which is 3.66 points lower than TSA_Part alone (56.28), while unseen mIoU rises from 36.67 to 40.51. The text states that applying guidance at both levels gives 'further improvements, especially in unseen classes,' but it does not mention the clear regression on seen classes. This trade-off is important because it indicates the object-level guidance may hurt the very classes used for training, and it is not explained. If the full model is used, the paper should discuss the reason or adjust the design; otherwise, the claim that both-level guidance is beneficial is unsupported.","section":"Sec. 4.3, Table 6"},{"comment":"The main results tables report single-run numbers without standard deviations or significance tests, and on ADE20K-Part-234 Pred-All the only existing baseline is PartCLIPSeg, whose seen mIoU is only 14.15. This makes the 12.81-point h-IoU claim for that benchmark rest on a single comparison point and on one trial. The difference between the two Lcomp variants in Table 5 (2.49 points in Pred-All h-IoU) may be within run-to-run noise. I ask the authors to provide at least three seeds with mean +/- std for the main tables and for the two ablation tables, and to add any available Pred-All numbers for PartGLEE and VLPart on ADE20K-Part-234, or to qualify the 'state of the art' claim for that setting.","section":"Sec. 4.2, Table 2 and general"}],"minor_comments":[{"comment":"The symbol FObj-Part is used both for the concatenation-projection output and, later in the same paragraph, for the convolved object-aware cost feature; please use distinct names (e.g., F^init and F^conv) to avoid confusion.","section":"Sec. 3.4, Eq. (8)"},{"comment":"The L1-normalization variant is not defined in the text; please give the exact form of Lcomp-L1 (e.g., normalization by the sum of absolute values rather than softmax) so the comparison is reproducible.","section":"Sec. 4.3, Table 5"},{"comment":"There is a duplicate citation '[8, 8]' in the list after 'Part Segmentation', and the dataset name is inconsistently spelled as 'PascalPart' in Section 4.1.","section":"Sec. 2 and Sec. 4.1"},{"comment":"The main text says the OOD split has 109 training/19 validation/30 test classes, but Table A7 lists only 109 base and 19 novel object classes; please reconcile the numbers and specify which 30 classes are used for test.","section":"Appendix F.4"},{"comment":"The comparison 'applying structural guidance at the part level yields more h-IoU increases than at the object level' should also state that the object-level-only row already includes the object-specific part-level guidance, per the table note, otherwise the attribution is unclear.","section":"Sec. 4.3, final paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is a direct continuation of the authors' PartCLIPSeg, and the architectural novelty is incremental; however, the empirical improvement is large and the code is public. The main concerns are not novelty but the robustness of the compositional loss under incomplete annotation and the lack of statistical grounding; both are fixable within the scope of a revision. The ADE20K Pred-All comparison with a single baseline should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a credible, well-scoped piece of work that will be useful to anyone working on open-vocabulary part segmentation. The core idea is to separate object-level and part-level cost volumes in a CAT-Seg-style pipeline, fuse them into an object-specific part volume, add a Jensen-Shannon loss that enforces part distributions to sum to the object distribution, and inject DINOv2 features into spatial aggregation. The components are individually known, but the combination is new and the ablations in Tables 5 and 6 give it internal support. The code is public, which I treat as real evidence.\n\nThe main results are large: +15 h-IoU on Pascal-Part-116, +12.8 on ADE20K-Part-234, +27.8 on PartImageNet in Pred-All. Those numbers come with caveats. The Pred-All baseline pool is thin, especially on ADE20K, where PartCLIPSeg is the only prior method and its seen mIoU is suspiciously low. The paper also does not report error bars or significance tests, so the gain on unseen classes could be less robust than it looks.\n\nThe stress-test worry about the compositional loss is legitimate and worth taking seriously, but it is not fatal. Equation (13) assumes every pixel has a single true parent object and that the part set is exhaustive. When annotation is partial, the loss can push part predictions to fill unannotated regions. The paper doesn't test this by masking unannotated pixels or by varying the mapping M. That is a real gap. But the ablation shows the loss helps, and the proposed mechanism is plausible; the right fix is a targeted experiment, not rejection.\n\nOther soft spots: reference [3] cites a cophylogenetic-analyis PACO paper where a part-segmentation citation is needed; that's sloppy and should be corrected. The authors do not disclose that PartCLIPSeg is their own prior work, which matters for context even though self-citation is not itself a flaw. And the Limitations appendix honestly admits it cannot handle instance-level part distinctions, so the claims stay within the semantic-segmentation framing.\n\nOverall: the paper is solid, the math is straightforward, the empirical work is extensive, and the code is available. I'd send it to review; a serious referee should demand the masked-loss ablation and better baseline coverage in Pred-All, but the central contribution holds up.","headline":"PartCATSeg is a strong empirical extension of CAT-Seg/PartCLIPSeg with clean ablations and code, but the headline gains rest on one protocol with thin baselines; the compositional-loss mechanism deserves a targeted stress test before I'd trust the numbers fully.","tokens_in":25273,"tokens_out":1656,"would_cite":true,"duration_ms":21218,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that separating object-level and part-level cost volumes, with a compositional loss and DINO structural guidance, sets a new state of the art for open-vocabulary part segmentation.","keywords":["open-vocabulary part segmentation","cost aggregation","image-text correspondence","compositional loss","DINO structural guidance","zero-shot segmentation","CLIP embeddings","harmonic IoU"],"falsifier":"Take a trained PartCATSeg model and randomly permute the part-to-object mapping used in the compositional loss during fine-tuning while keeping all other components fixed; if the harmonic-mean IoU on unseen parts does not drop, the compositional loss's claimed mechanism is not the source of the gains. Alternatively, train on a dataset with artificially incomplete part annotations (e.g., dropping a subset of part masks) and compare with and without the compositional loss.","tokens_in":24153,"feed_emoji":"🧩","tokens_out":2843,"duration_ms":29049,"temperature":0.7,"pith_summary":"Open-vocabulary part segmentation aims to label fine-grained object parts for categories never seen during training, but part-level image-text alignment is weak and models lack structural understanding of how parts relate to objects. This paper proposes PartCATSeg, which builds on cost aggregation to explicitly compute and refine separate cost volumes for objects and parts, then fuses them into object-specific part costs. It adds a compositional loss that enforces the inductive bias that parts collectively compose their object, compensating for scarce part annotations, and injects DINO features as structural guidance during aggregation. The paper reports large improvements over prior methods on Pascal-Part-116, ADE20K-Part-234, and PartImageNet, with the largest gains on unseen classes. If these results hold, PartCATSeg provides a new baseline for robust generalization in open-vocabulary part segmentation.","feed_headline":"Part segmentation model leaps up to 27.8 points on unseen classes","feed_subtitle":"Disentangling object and part costs plus a compositional loss sets a new open-vocabulary part baseline.","key_machinery":"The central object is the disentangled cost aggregation architecture: two separate image-text cost volumes, one for object class names and one for part class names, are each refined by spatial- and class-aggregation transformers before being concatenated and projected into an object-specific part cost volume that is refined again. The compositional loss is the key identity: at each pixel, the softmax distribution over object classes is compared by Jensen-Shannon divergence to the distribution obtained by summing part-class probabilities according to a fixed mapping from each part to its object, injecting the prior that parts compose the whole. Structural guidance is the third mechanism: DINO's self-supervised features are fed into the query and key of the spatial aggregation transformers, supplying geometric and boundary information that complements CLIP's semantic alignment.","core_discovery":"PartCATSeg establishes that disentangling the image-text matching signal by object and part levels, rather than pooling them into one cost volume, markedly improves fine-grained part alignment. Three mechanisms carry the result: separate spatial and class aggregation transformers refine an object cost volume and a part cost volume independently; a projection fuses the refined features into an object-specific part cost volume aligned with names like 'bird's head'; and a compositional loss forces the softmax distribution over object classes to match the sum of part-class probabilities mapped through a fixed part-to-object correspondence. DINO features are appended to the cost volume during spatial aggregation, providing geometric structure that helps delineate part boundaries. On the three main benchmarks the method reports harmonic-mean IoU gains of 15.10, 12.81, and 27.79 points over the second-best method in the Pred-All setting, with the largest relative improvements on unseen part classes.","pith_inferences":["The fixed part-to-object mapping in the compositional loss could likely be replaced by a learned or probabilistic mapping, which may extend the method to datasets where part classes are shared across object categories.","The disentangled cost-volume design might transfer to other fine-grained recognition problems, such as attribute segmentation or human parsing, where object context and fine-grained labels have a similar part-whole hierarchy.","A testable extension is to apply the same three mechanisms to instance-level part segmentation by combining them with an off-the-shelf open-vocabulary instance segmentation module, which the authors themselves flag as future work.","The reported ablation suggests that the structural guidance is more useful at the part level than the object level, implying that the benefit comes from intra-object geometry rather than background separation."],"forward_implications":["If the reported gains replicate, cost aggregation becomes a validated mechanism for fine-grained open-vocabulary tasks beyond whole-object segmentation.","The compositional loss provides a template for leveraging scarce part annotations by transferring supervision from object-level signals to part-level predictions.","DINO features, already known for semantic correspondence, are shown to be directly useful inside an image-text cost volume for improving part boundaries.","The Pred-All evaluation numbers suggest that full part segmentation without any object-mask oracle is closer to being practical, since unseen-class performance is no longer far below seen-class performance."],"supporting_citations":[{"why":"Supplies the base cost aggregation architecture (spatial and class aggregation transformers, fine-tuned CLIP encoders) that PartCATSeg extends to object and part levels.","marker":"[13]"},{"why":"Defines the Pred-All evaluation protocol and the strongest prior baseline (PartCLIPSeg) that PartCATSeg must outperform on unseen classes.","marker":"[14]"},{"why":"Provides the Oracle-Obj evaluation setting and the adapted OVPS versions of Pascal-Part and ADE20K datasets used in the experiments.","marker":"[70]"},{"why":"Supplies the DINOv2 features whose structural guidance is injected into the spatial aggregation transformers.","marker":"[54]"},{"why":"Provides the frozen CLIP image and text encoders that produce the dense visual and language embeddings forming the cost volumes.","marker":"[57]"},{"why":"Establishes the transformer-based cost aggregation machinery that CAT-Seg and PartCATSeg build upon for matching.","marker":"[11]"}],"fun_headline_variants":["Disentangling image-text costs yields 27.8-point part segmentation gain","Open-vocab part segmentation: disentangled costs leap 27.8 points","PartCATSeg: splitting costs and DINO guidance redefine part segmentation","Object-part cost split boosts unseen class segmentation by 27.8 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The compositional loss assumes a fixed, correct mapping from each part class to its object class; if training masks omit some parts, part classes are shared across objects, or the mapping is mis-specified, the enforced part-to-object consistency could distort rather than sharpen part predictions.","fun_headline_variants_meta":{"raw":{"variants":["Disentangling image-text costs yields 27.8-point part segmentation gain","Open-vocab part segmentation: disentangled costs leap 27.8 points","PartCATSeg: splitting costs and DINO guidance redefine part segmentation","Object-part cost split boosts unseen class segmentation by 27.8 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2840,"prompt_tokens":908,"completion_tokens":1932,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1851}},"tokens_in":524,"tokens_out":1932,"duration_ms":13424,"temperature":1.0,"reasoning_tokens":1851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:45:29.420380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained PartCATSeg model and randomly permute the part-to-object mapping used in the compositional loss during fine-tuning while keeping all other components fixed; if the harmonic-mean IoU on unseen parts does not drop, the compositional loss's claimed mechanism is not the source of the gains. Alternatively, train on a dataset with artificially incomplete part annotations (e.g., dropping a subset of part masks) and compare with and without the compositional loss.","supporting_citations":[{"cited_title":"Understanding Multi-Granularity for Open-Vocabulary Part Segmentation","cited_arxiv_id":"2406.11384","evidence_quote":"Defines the Pred-All evaluation protocol and the strongest prior baseline (PartCLIPSeg) that PartCATSeg must outperform on unseen classes."},{"cited_title":"Ov-parts: Towards open- vocabulary part segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the Oracle-Obj evaluation setting and the adapted OVPS versions of Pascal-Part and ADE20K datasets used in the experiments."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP image and text encoders that produce the dense visual and language embeddings forming the cost volumes."},{"cited_title":"Cats: Cost aggre- gation transformers for visual correspondence","cited_arxiv_id":null,"evidence_quote":"Establishes the transformer-based cost aggregation machinery that CAT-Seg and PartCATSeg build upon for matching."}],"review_version":1}