{"id":"a8b723c0-a540-4467-9e4a-d3cd703b1866","arxiv_id":"2412.12740","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Con2MAV discovers new semantic classes and object instances at test time, with the new PANIC benchmark for open-world panoptic segmentation.","lead":"This paper introduces Con2MAV, a network that segments both known and never-seen objects and classes in images, plus PANIC, a new driving dataset for testing such open-world segmentation. The method is evaluated on several benchmarks with competitive results, although the code is not yet released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PANIC test metrics mix known and unknown unknowns; aggregate results cannot substantiate discovery of truly novel categories, leaving the void-area training assumption untested.","rationale":"The reader's weakest assumption identifies the reliance on Cityscapes void areas as the training signal for unknowns and worries about miscalibration. This stress-test agrees that this reliance is the most load-bearing condition for the central claim, but identifies a more precise and actionable issue: the PANIC hidden test set is a mixture of known unknowns and unknown unknowns, and the paper reports only aggregate metrics. Because the known unknowns are drawn from the very distribution used to train the objectosphere loss, their presence in the test set can inflate the aggregate scores and mask poor performance on the unknown unknowns—the categories that actually demonstrate open-world discovery. This is not merely a theoretical miscalibration worry; it is a missing analysis that can be performed with the ground truth the authors already possess. The paper honestly discloses the void-area limitation in Section 7, which is creditable, but the experimental section does not isolate the effect. The verification step is concrete and low-cost: split the hidden test set metrics by class type. If the two groups perform similarly, the concern is resolved; if not, the central claim of discovering genuinely novel categories is unsupported. The reader's conditional verdict (CONDITIONAL) remains appropriate: acceptance should require this per-split evidence, or at least an explicit statement that aggregate scores are not sufficient to demonstrate generalization to unknown unknowns. No other concern appears more load-bearing: the offline instance decoder is at least evaluated on COCO open-set splits with competitive results, and the permissive metric is mitigated by supplementary homogeneity/completeness scores. The contradiction between 'we do not rely on unknown objects in the training set' (Related Work) and Section 7's admission is real but secondary; it primarily affects framing, not the technical argument. The proposed check targets the core claim directly.","tokens_in":29305,"tokens_out":8882,"duration_ms":88750,"concrete_test":"On the PANIC hidden test set ground truth (available to the authors), recompute all open-world semantic and panoptic metrics—mIoU, completeness, homogeneity, PQ, RQ, SQ—separately for the validation classes (known unknowns that appear in Cityscapes void) and the test-only classes (unknown unknowns with no Cityscapes presence). If the test-only metrics are comparable to the validation-class metrics, the void-area training signal generalizes to true novel categories. If the test-only metrics are substantially lower, the aggregate results overstate the method's open-world discovery capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Con2MAV discovers novel semantic categories and instances at test time rests on the method's ability to recognize objects that were entirely absent from Cityscapes training, including the void/unlabeled areas used for the objectosphere loss (Eq. 8, Sec. 4.2). Section 7 explicitly identifies reliance on these void areas as the main limitation. PANIC's validation set is deliberately restricted to 'known unknowns'—classes that appear in Cityscapes void—while the hidden test set contains both validation and test classes, the latter being true 'unknown unknowns' such as scooters and forklifts (Sec. 5). Tables 8 and 12 report only aggregate metrics on the hidden test set. Because the aggregate mixes easy known-unknown categories (which directly match the training-time void distribution) with hard unknown-unknown categories, the reported mIoU and PQ could be dominated by the easy classes. The paper therefore does not separately evidence the load-bearing sub-claim that the method discovers truly novel categories beyond the void distribution. The reader's weakest assumption is exactly this void-area dependency; the present concern sharpens it into a concrete, observable gap in the reported evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Con2MAV, a fully convolutional network for open-world panoptic segmentation that extends the authors' previous ContMAV architecture. The method uses three decoders (semantic, contrastive, and instance) to segment known classes, detect unknown regions, discover novel semantic categories at test time, and segment instances within those categories. The paper also introduces PANIC, a new benchmark dataset for open-world segmentation in autonomous driving, with 800 images, 58 unknown classes, 4000+ instances, and public competitions for four tasks: anomaly segmentation, open-world semantic segmentation, open-set panoptic segmentation, and open-world panoptic segmentation. Experiments on SegmentMeIfYouCan, BDDAnomaly, COCO, SUIM, and PANIC report state-of-the-art results on several open-world tasks while retaining competitive closed-world performance. The paper also proposes a standardized nomenclature for open-world segmentation tasks.","tokens_in":29558,"tokens_out":7079,"duration_ms":59870,"significance":"If the results hold, the paper makes two valuable contributions. First, it extends open-world semantic segmentation to the panoptic setting, which is a natural and important step for autonomous perception. Second, the PANIC dataset and its public competitions address a real gap in benchmarks: existing anomaly datasets either lack semantic/instance annotations or have limited class diversity. The paper's approach is lightweight and the authors provide ablation studies showing the contribution of the pre-logit feature space and the instance-decoder losses. The explicit discussion of limitations and the attempt to unify task nomenclature are also useful. However, the evaluation has gaps that weaken the central claim of discovering truly novel categories, so the current version requires revision.","major_comments":[{"comment":"The PANIC validation set contains only 'known unknowns' (classes present in the Cityscapes void areas), while the hidden test set mixes these with 'unknown unknowns'; the paper reports only aggregate metrics on the hidden test set. Because the objectosphere loss (Eq. 8, Sec. 4.2) is trained on Cityscapes void areas, the learned anomaly detector is calibrated to the known-unknown distribution, and the aggregate mIoU/PQ figures could be dominated by the easy classes. The central claim that Con2MAV discovers truly novel categories at test time therefore needs separate results for the unknown-unknown subset (or per-class results) to be substantiated; the authors should report this breakdown or explicitly qualify the claim.","section":"5, Tables 8 and 12; Section 7"},{"comment":"For the newly introduced task of open-world panoptic segmentation, the paper reports only Con2MAV results and no comparative baseline. A simple baseline such as ContMAV (with an instance decoder or with the proposed clustering) or Mask2Anomaly would contextualize the 24.3% PQ and the claimed 'first approach' status. The absence of a baseline makes it impossible to assess whether the proposed modules are necessary for the task.","section":"6.6, Table 12"},{"comment":"The proposed open-world IoU metric matches each predicted class to its best ground-truth class via argmax(row_i), which is permissive and rewards methods that over-segment. The paper does report homogeneity and completeness, but the headline mIoU numbers in Tables 6-9 and the state-of-the-art claims rely on this metric. The authors should add a stricter matching (e.g., Hungarian matching or a discussion of the effect of the argmax matching) and report the resulting numbers, or at least justify the choice with an analysis.","section":"5.2.2"},{"comment":"All reported results are single runs without error bars or significance tests. Given the stochasticity of training and the clustering post-processing, the claimed improvements (e.g., 'outperform ContMAV by 19% mIoU' on SUIM) may not be stable. At minimum, the main comparisons should include multiple seeds with standard deviations, or the authors should state that the differences are within run-to-run variability.","section":"6, all tables"}],"minor_comments":[{"comment":"The phrase 'object that have never been seen' should be 'objects that have never been seen'; also 'segmentaton' in the Introduction is a typo.","section":"Abstract"},{"comment":"The definition of the open-world semantic mask Ms uses 0 for 'known' and 1..K for discovered categories, but the rest of the paper (e.g., Eq. (1), Table 2) uses 1..K for the known classes; this notation conflict should be resolved.","section":"3.2"},{"comment":"The notation e_p = p is confusing; it may be clearer to define the offset prediction o_p and write e_p = p + o_p.","section":"4.2, Eq. (11)"},{"comment":"The last two columns of the ablation table are not labeled; the reader has to infer that they are mIoU and PQ.","section":"Table 13"},{"comment":"Typo: 'evaluted' should be 'evaluated'.","section":"5.2.2"},{"comment":"Typo: 'intersting' should be 'interesting'.","section":"7"},{"comment":"Typo: 'extpected' should be 'expected'.","section":"Supplementary Fig. 9"},{"comment":"The 'OoD' column header is not defined in the caption; define it.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is likely to be the paper's most lasting impact, and the public competitions are a strong addition. The method itself is an incremental extension of ContMAV, and the evaluation gaps (known-unknown bias, no baselines on the new task, permissive metric) need to be addressed before the SOTA claims can be accepted. I suggest the editors weigh the dataset and benchmark contributions separately from the method claims in the revision decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Real contribution here is the benchmark, not the method. PANIC is a new open-world panoptic segmentation dataset with 800 images, 58 unknown classes, 4000 instances, a hidden test set, and public competitions. That is useful and well thought out. The task definitions and proposed nomenclature in Section 3 also help clear up a confusing area. On the method side, Con2MAV is an honest extension of the authors' ContMAV: pre-logit descriptors instead of pre-softmax, an instance decoder with offset losses, and learned thresholds. The SUIM experiments show a real improvement when few known classes are available, which was a documented weakness of ContMAV. Ablations are informative.\n\nThe soft spots are in the evaluation, not in the math. The hidden test set for PANIC contains both known unknowns (classes that appear in Cityscapes void areas) and true unknown unknowns (e.g., scooters, forklifts). All reported test-set metrics are aggregate. That means the reader cannot tell whether the method actually discovers genuinely novel categories, or just recognizes things similar to the void-area training signal. The paper itself states in Section 7 that the objectosphere loss relies on void/unlabeled areas. So the dependence is acknowledged; what is missing is a per-class breakdown on the test set that separates known-unknown from unknown-unknown performance. Without that, the central claim of 'discovering novel semantic categories' is not substantiated. The stress-test concern lands squarely here.\n\nOther issues are more minor. No error bars anywhere. The open-world IoU metric uses argmax matching, which is permissive by design but worth flagging. No baselines on the new open-world panoptic task—only the method's own numbers, and the comparison is to previous ContMAV on related tasks. Code is promised but not released. None of these are fatal, but they all push in the same direction: the paper needs to show its work more.\n\nWho gets value from this? Anyone working on open-world perception, anomaly segmentation, or autonomous driving benchmarks. The benchmark alone justifies a serious referee. The method is coherent and the writing is clear. I would send it to review, but with the expectation of major revision focused on disaggregating the PANIC test set results and adding or acknowledging the missing baselines.","headline":"PANIC is a genuinely useful benchmark and the task formulation helps the field, but the paper's headline claim about discovering novel categories is not yet backed by its own evaluation.","tokens_in":30051,"tokens_out":2702,"would_cite":true,"duration_ms":24911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single fully convolutional network can, at test time, discover new semantic categories and segment individual instances of them, while keeping competitive accuracy on its training classes.","keywords":["open-world panoptic segmentation","anomaly segmentation","novel class discovery","class descriptors","contrastive learning","instance segmentation","benchmark dataset","autonomous driving"],"falsifier":"Train the same architecture on Cityscapes after removing all unlabeled background pixels from the objectosphere loss (or on a fully labeled dataset with no such background), evaluate on the PANIC hidden test set, and compare anomaly AUPR and unknown-class mIoU: if performance collapses, the background signal is load-bearing; if it holds, the method does not actually need unlabeled training pixels.","tokens_in":29110,"feed_emoji":"🚗","tokens_out":7695,"duration_ms":63987,"temperature":0.7,"pith_summary":"This paper introduces the task of open-world panoptic segmentation: at test time, a vision system should not only flag objects it has never seen but also group them into new semantic categories and separate them into individual instances, all without retraining. The authors propose Con2MAV, a fully convolutional network with three decoders, and claim it is the first approach and the first benchmark with a public competition for this task. Con2MAV builds a fixed-dimensional descriptor for every class it was trained on, then uses those descriptors to open new classes when novel objects appear; a separate instance decoder locates each object within the anomalous area. The paper reports strong results across five datasets, covering driving, everyday, and underwater scenes, while keeping closed-world accuracy close to a model trained without any open-world component. It also releases PANIC, an 800-image driving benchmark with more than 50 unknown classes and over 4000 instances, with a hidden test set.","feed_headline":"A network that discovers new classes and objects at test time","feed_subtitle":"It segments never-seen objects into new classes and instances; a new 800-image benchmark tests it.","key_machinery":"The load-bearing object is the pre-logit class descriptor. Before the final convolution, every known class $k$ is represented by a running mean $\\mu_k$ and variance $\\sigma_k^2$ in a fixed-dimensional feature space $\\mathbb{R}^D$, built from true-positive pixels only (Eqs. 2–3). The feature loss (Eq. 4) pulls each pixel's pre-logit toward its class's descriptor; the contrastive loss (Eq. 7) scatters class means on the unit sphere; and the objectosphere loss (Eq. 8) drives the norm of known-class features above 1 and unknown features toward 0. At test time, the squared exponential kernel (Eq. 18) scores a pixel's fit to each known Gaussian, and a $1\\sigma$ bound decides \"unknown\"; the first unknown pixel seeds a new descriptor that is updated with a running mean and variance, so newly discovered classes stay consistent across images. The instance decoder's offset predictions are clustered with HDBSCAN only inside \"thing\" areas, and the semantic prediction filters clusters so pixels from different classes are not merged.","core_discovery":"The central claim is that discovering new semantic classes and new object instances at test time does not require out-of-distribution training data, generative models, or language models; a fully convolutional architecture with carefully chosen losses suffices. During training, the semantic decoder accumulates a running mean and variance of pre-logit features for each known class (Eqs. 2–3) and a feature loss pulls true-positive pixels toward those descriptors (Eq. 4). The contrastive decoder pushes known-class features onto the unit hypersphere while driving void or unlabeled feature norms toward zero (Eqs. 7–8), and the instance decoder predicts per-pixel offsets toward each instance centroid using a Lovász hinge plus divergence and curl regularizers (Eqs. 10–16). At test time a pixel is unknown only if it fails both a $1\\sigma$ Gaussian-fit test against the known descriptors and a $1\\sigma$ norm test from the contrastive decoder; the first unknown pixel's descriptor seeds a new class that evolves via a running average, and HDBSCAN clusters the offset predictions inside \"thing\" areas, with the semantic prediction filtering the resulting instances. The paper argues that this coupling enables consistent class discovery and instance segmentation together, and it reports top results on SegmentMeIfYouCan, BDDAnomaly, COCO, SUIM, and PANIC.","pith_inferences":["The training signal for \"unknown\" is the unlabeled background pixels of Cityscapes; a deployment site whose novel objects look nothing like those background objects would likely see miscalibrated anomaly scores unless the detector is re-calibrated or augmented with out-of-distribution data.","Nothing in the method is specific to cameras: the losses and post-processing apply to any per-pixel feature backbone, so a LiDAR or RGB-D variant is a natural next step that the paper itself mentions as future work.","The open-world semantic evaluation matches each predicted class to the ground-truth class it overlaps most, which rewards purity but does not penalize splitting one true class into several predicted classes; adding a split-penalty metric would sharpen comparisons.","Because new classes are seeded from a single pixel's descriptor, an anomalous pixel that is unrepresentative of its category could create a noisy prototype; seeding classes from a small consensus cluster of mutually close anomalous descriptors is a testable robustness improvement."],"forward_implications":["A robot or vehicle can flag, categorize, and separate objects it has never seen without retraining or extra data, which is the main safety-relevant capability the paper targets.","The same training recipe transfers across domains: experiments on Cityscapes for driving, COCO for everyday scenes, and SUIM for underwater imagery all report competitive open-world and closed-world numbers.","Discovered classes persist: once an anomaly seeds a new descriptor, later appearances of the same category are matched to that descriptor, giving test-time consistency without incremental training.","The pre-logit design removes the fragile, manually tuned thresholds of the predecessor and also fixes the few-known-classes failure mode, shown by a roughly 19% mIoU gain on SUIM.","PANIC provides the first public, hidden-test-set benchmark for open-world panoptic segmentation in autonomous driving, with competitions for all four open-world tasks."],"supporting_citations":[{"why":"The predecessor ContMAV whose feature-loss machinery and thresholds Con2MAV extends; all open-world semantic segmentation baselines compare against it.","marker":"[75]"},{"why":"Cityscapes defines the known classes and the unlabeled background areas that supply the training signal for unknown.","marker":"[16]"},{"why":"SegmentMeIfYouCan provides the anomaly segmentation benchmark and the pixel- and component-level metrics used throughout.","marker":"[10]"},{"why":"The objectosphere loss that drives unknown feature norms toward zero in the contrastive decoder.","marker":"[19]"},{"why":"The supervised contrastive loss that scatters class means on the unit hypersphere.","marker":"[11]"},{"why":"The divergence and curl vector-field losses used to regularize the instance-decoder offset predictions.","marker":"[83]"},{"why":"Mask2Anomaly is the main baseline for open-set panoptic segmentation on COCO and defines the COCO unknown-class splits.","marker":"[65]"},{"why":"CAOS and BDDAnomaly provide one of the open-world semantic and anomaly test sets, including baseline numbers from other methods.","marker":"[30]"}],"fun_headline_variants":["Open-world panoptic segmentation discovers unseen classes and instances","New benchmark and model for segmenting unknown objects in driving scenes","Con2MAV: Finds new classes and objects at test time without retraining","Segmentation that spots never-seen objects and labels them on the fly","PANIC benchmark: 800 images of unknown objects for open-world segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training signal for what counts as unknown comes from the unlabeled background areas of the Cityscapes images; if real-world anomalies look different from those background objects, the anomaly scores and new-class descriptors will be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Open-world panoptic segmentation discovers unseen classes and instances","New benchmark and model for segmenting unknown objects in driving scenes","Con2MAV: Finds new classes and objects at test time without retraining","Segmentation that spots never-seen objects and labels them on the fly","PANIC benchmark: 800 images of unknown objects for open-world segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1486,"prompt_tokens":1117,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":733,"tokens_out":369,"duration_ms":3796,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:46:28.796653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same architecture on Cityscapes after removing all unlabeled background pixels from the objectosphere loss (or on a fully labeled dataset with no such background), evaluate on the PANIC hidden test set, and compare anomaly AUPR and unknown-class mIoU: if performance collapses, the background signal is load-bearing; if it holds, the method does not actually need unlabeled training pixels.","supporting_citations":[{"cited_title":"Open- world semantic segmentation including class similarity,","cited_arxiv_id":null,"evidence_quote":"The predecessor ContMAV whose feature-loss machinery and thresholds Con2MAV extends; all open-world semantic segmentation baselines compare against it."},{"cited_title":"The Cityscapes dataset for semantic urban scene understanding,","cited_arxiv_id":null,"evidence_quote":"Cityscapes defines the known classes and the unlabeled background areas that supply the training signal for unknown."},{"cited_title":"Reducing network agnosto- phobia,","cited_arxiv_id":null,"evidence_quote":"The objectosphere loss that drives unknown feature norms toward zero in the contrastive decoder."},{"cited_title":"Panoptic Segmentation with Partial Annotations for Agricultural Robots,","cited_arxiv_id":null,"evidence_quote":"The divergence and curl vector-field losses used to regularize the instance-decoder offset predictions."},{"cited_title":"Mask2anomaly: Mask transformer for universal open-set segmentation,","cited_arxiv_id":null,"evidence_quote":"Mask2Anomaly is the main baseline for open-set panoptic segmentation on COCO and defines the COCO unknown-class splits."},{"cited_title":"Scaling out-of-distribution detection for real- world settings,","cited_arxiv_id":null,"evidence_quote":"CAOS and BDDAnomaly provide one of the open-world semantic and anomaly test sets, including baseline numbers from other methods."}],"review_version":1}