{"id":"e945b507-b682-4fb7-854a-94b29a027a81","arxiv_id":"2412.06470","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OREAL improves patch-based active learning for semantic segmentation by scoring superpixels with their maximum pixel uncertainty and using one-vs-rest entropy to balance classes.","lead":"OREAL is a new active learning method for image segmentation that selects the most informative image patches by focusing on the most uncertain boundary pixels, rather than averaging uncertainty across the patch. In tests on four datasets and three model types, it gave small but consistent improvements in segmentation accuracy with the same annotation budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OREAL's implicit class balancing via OVR entropy is the weakest link: Algorithm 1 may select superpixels whose true label is not the intended class, and the paper never isolates OVR from max aggregation, so the central claim that OVR adds measurable value is unsupported.","rationale":"After reading the paper in good faith, the most load-bearing assumption in the central claim is not the max-aggregation hypothesis per se—that component is empirically supported by the paper's consistent mean-vs-max comparisons (Table 1, Fig. 4) and by the qualitative boundary-selection examples. The load-bearing assumption is that OVR entropy actually achieves implicit class balancing and that this contributes to performance. This assumption is structurally fragile: Eq. (2) scores a superpixel by a binary-entropy maximum, which peaks at Pi[c]=0.5 regardless of the true class, so selecting by Hc does not guarantee the selected superpixel belongs to class c. The paper's own Sec. 3.5 admits this but provides no evidence that the deviation self-corrects. If the deviation is systematic (e.g., rare classes rarely have Pi[c] near 0.5 in true positives), the labeled set will remain imbalanced and the method degenerates to max-entropy with a per-class budget. No experiment reported in the paper isolates this component: Table 1 and Fig. 4 vary aggregation and compare to baselines, but never compare OREAL to a version with the same δc budget and max entropy instead of OVR. Therefore the central claim that OVR is a beneficial novelty is unverified. My proposed concrete test directly separates the OVR score from the budget mechanism and measures the actual class distribution, which would settle the concern. This aligns partially with the reader's 'weakest_assumption': both are about the reliability of a proxy (max pixel vs OVR-as-balancing), but the reader focused on max while I focus on the class-balancing mechanism, which is the part the paper claims as new.","tokens_in":12896,"tokens_out":4465,"duration_ms":44450,"concrete_test":"Run an ablation on Pascal VOC or Cityscapes: (A) OREAL as published; (B) max-aggregated categorical entropy with the same per-class budget δc from Eq. (3), i.e., select top-δc superpixels by max entropy per class; (C) OREAL but log, at each iteration, the true class labels of selected superpixels versus δc. Compare AuALC across 3 runs. If (A) does not significantly exceed (B), the OVR score adds no measurable value; if the per-class deviation for tail classes exceeds roughly 20% of δc, the implicit balancing claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty beyond max aggregation is the one-vs-rest (OVR) entropy for implicit class balancing (Abstract, Sec. 3.3-3.5). In Algorithm 1 (Sec. 3.5), for each class c the method selects the δc superpixels with highest Hc[P] = max_{i∈K} Hc[i], where Hc[i] = -Pi[c] log Pi[c] - (1-Pi[c]) log(1-Pi[c]). The problem: Hc[i] is maximized when Pi[c] ≈ 0.5, irrespective of whether the true label is c. For a rare class c, a superpixel belonging to a frequent class d may have Pi[c] ≈ 0.5 due to confusion with c, and thus be selected as a 'class c' sample even though its true label is d. The paper acknowledges this in Sec. 3.5 ('uncertain samples of class c do not guarantee that they are actually from that class') but only asserts that deviation is compensated in subsequent iterations, without any analytical argument or empirical verification. If compensation fails, the actual labeled set does not achieve the δc distribution, and the claimed implicit class balancing does not occur. Furthermore, the experimental section never isolates OVR: Table 1 compares full OREAL against other methods, and the ablation in Fig. 4 varies mean/max but not the OVR component. A comparison of OREAL against max-aggregated plain entropy with the same δc distribution would be needed to show OVR itself (rather than max aggregation) is responsible for any gain. Without this, the central claim that OREAL's class-wise score provides a benefit over existing entropy-based sampling is unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OREAL, a patch-based active learning method for semantic segmentation. It makes two main contributions: (i) replacing mean aggregation of pixel-wise uncertainty scores with maximum aggregation when scoring superpixels, in order to sample boundary regions, and (ii) introducing one-vs-rest (OVR) entropy, a per-class uncertainty score intended to implicitly balance the class distribution of the selected annotation set. The method is evaluated on four datasets (Cityscapes, Pascal VOC, MONARCH, EndoVis) with three backbones (ResNet-50, ResNet-101, ViT), against several baselines, using the AuALC metric over six active learning steps. The experiments show that on average, maximum aggregation improves results over mean aggregation, and that OREAL is competitive with or better than baselines on most datasets, except for MONARCH, where OREAL is generally outperformed. The paper also presents an extension to weak labeling and an analysis of annotation cost to reach 95% of full-training performance.","tokens_in":13242,"tokens_out":4176,"duration_ms":43241,"significance":"If the claims hold, the paper would make a useful practical contribution to active learning for semantic segmentation. The maximum-aggregation idea is simple, easy to implement, and appears to yield consistent average gains across diverse datasets and architectures, as evidenced by the paired mean/max comparisons in Table 1. The OVR entropy score is a novel formulation, and the paper includes a thorough experimental protocol with multiple runs and several datasets. The explicit acknowledgement of the MONARCH counterexample and the analysis of superpixel dominant-label noise are also strengths. However, the significance is tempered by two gaps: the OVR component is never isolated experimentally, so the paper's central novelty beyond maximum aggregation is not directly supported, and the claimed implicit class balancing rests on an unverified assumption about the reliability of per-class uncertainty selection. These issues are fixable with additional experiments and analysis, and the paper's core hypothesis about maximum aggregation is largely supported by the data.","major_comments":[{"comment":"The contribution of the one-vs-rest entropy is never isolated. Table 1 compares full OREAL against baselines, and Fig. 4 ablates mean vs. max aggregation for all methods, but there is no variant of OREAL that uses maximum aggregation with a standard entropy (or another per-class score) under the same per-class quota δc. Without such an ablation, the observed gains cannot be attributed to OVR entropy rather than to the max aggregation or to the explicit quota mechanism in Algorithm 1. Since the class quota is already enforced by steps 6–10 of Algorithm 1, the per-class balancing is achieved by the quota itself; OVR entropy only selects within each quota. The paper should either provide a direct comparison of OVR entropy with a non-OVR score under identical quota conditions, or substantially soften the claim that the OVR score provides a distinct benefit.","section":"Sec. 3.3/3.5, Algorithm 1, Table 1, Fig. 4"},{"comment":"The statement that 'any deviation from δc is compensated for in subsequent iterations' is unsupported. Algorithm 1 selects the top-δc superpixels by predicted class c, but the true dominant label of a selected superpixel may differ from c, as the paper itself acknowledges. The claimed compensation is deferred to an Appendix that is not included in the manuscript. The paper should provide empirical evidence on the fraction of selected superpixels whose true label matches the requested class, and demonstrate that the actual labeled class distribution converges to the balanced target over iterations. Without this, the 'implicit class balancing' claim is not established.","section":"Sec. 3.5, Algorithm 1"},{"comment":"The experimental support for the central claim is weakened by the absence of error bars or significance tests in Table 1, despite the paper reporting multiple runs. Fig. 4 shows error bars, but Table 1 is used for the main cross-method comparisons. Moreover, the MONARCH dataset systematically contradicts the general claim: for example, OREAL max achieves 69.7 AuALC on MONARCH with ResNet-50, while Revisiting SP max achieves 73.7, and similar gaps appear with ResNet-101. The paper acknowledges this but does not analyze why OREAL fails on MONARCH. Given the paper's claim of comprehensive validation, the authors should provide a statistical comparison (e.g., paired tests with variance information) and a more detailed investigation of when max aggregation and OVR entropy help or hurt, including the MONARCH setting.","section":"Table 1, Fig. 4, Sec. 4.3"},{"comment":"The load-bearing assumption that the maximum pixel-wise uncertainty is a reliable proxy for a superpixel's annotation value is not examined. A single noisy or outlier pixel can dominate the max score, and the paper provides no analysis of how often this occurs or how sensitive the results are to this choice. A robustness study (e.g., comparing max with other extreme quantiles, or analyzing the label purity of selected superpixels) would strengthen the central claim. This is not a fatal flaw, but it is a significant missing analysis for a method whose entire rationale rests on the max operation.","section":"Sec. 3.2, Fig. 2"}],"minor_comments":[{"comment":"There are several typos and grammatical issues, for example 'we found out that using' in Sec. 3.2, 'A verage' in Table 1, and 'a novel uncertainty score function' in the Abstract. A careful proofreading pass is recommended.","section":"Throughout"},{"comment":"The caption states that error bars indicate one standard deviation, but Table 1, which is the primary quantitative summary, does not report any variance information. Please make the reporting consistent.","section":"Fig. 4 caption"},{"comment":"The sentence 'This labeling method is known as the dominant label' appears in Sec. 2.2 rather than Sec. 4.2; the notation for the split proportions is also a bit cumbersome (e.g., '4 : 1for EndoVis'). Consider reformatting for clarity.","section":"Sec. 4.2"},{"comment":"In the superpixel dominant labeling discussion, the sentence 'A model A is trained... and a model B is trained...' lacks the verb 'is' before 'trained' in the second clause; this is likely a typo.","section":"Sec. 5"},{"comment":"The paper repeatedly refers to an Appendix for details on the compensation mechanism and for additional results, but the submitted manuscript does not include the appendix. Please either include the appendix or remove the references to it.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical contribution on maximum aggregation is reasonably well supported, but the novelty around OVR entropy and implicit class balancing is not backed by a direct ablation. The lack of error bars in the main table and the unexplained MONARCH counterexample also undermine the generality of the claims. These issues are addressable with additional experiments and analysis, so I recommend major revision rather than rejection. The authors should be encouraged to provide the missing OVR ablation, report variance and significance, and give a more careful discussion of when their method works."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The max aggregation idea is real and well supported. Swapping mean for max when scoring superpixels improves boundary sampling and gives consistent gains across four datasets and three backbones, and the authors show it for the baselines as much as for their own method. That is a clean, useful observation any patch-based AL pipeline can adopt. The second contribution, one-vs-rest entropy, is the weak link. The paper never isolates it.\n\nAlgorithm 1 picks the top δc superpixels by Hc per class. Hc is binary entropy of the class probability, so it peaks at around 0.5 regardless of whether the true label is c. The selected superpixel is not guaranteed to belong to class c. The paper acknowledges this in Sec. 3.5 but only asserts that deviations are compensated later; there is no analysis and no empirical verification. More importantly, no experiment compares OREAL against max-aggregated plain entropy with the same per-class budget. Without that, the claim that OVR entropy improves results is unsupported. The stress-test note lands.\n\nOther soft spots, in proportion. Table 1 has no error bars or significance tests even though runs are averaged. MONARCH goes against the trend, though the authors give a reasonable explanation for why. No code is released, which hurts a methods paper. The reader's list is accurate.\n\nCredit where due. The experiments are extensive: four datasets, three backbones, two labeling schemes. The mean-vs-max ablation in Fig. 4 is the right design and makes the first contribution convincing. The context-sampling intuition is clearly explained, and the weak-labeling appendix is a nice extra.\n\nBottom line: the max aggregation result deserves attention and probably a citation. The OVR part needs real work before the paper can claim it. I would send this to peer review because the max result is valuable and the OVR issue is fixable, but a referee should require the OVR isolation ablation and a serious treatment of the class-balancing selection problem.","headline":"Max aggregation is a solid, simple win; the one-vs-rest entropy claim is not isolated and the class-balancing mechanism has a hole.","tokens_in":13776,"tokens_out":2906,"would_cite":true,"duration_ms":29652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing mean with maximum aggregation of pixel-wise uncertainty in superpixel scoring improves boundary sampling and segmentation quality, and that a one-vs-rest entropy score adds implicit class balancing.","keywords":["active learning","semantic segmentation","superpixel","uncertainty aggregation","one-vs-rest entropy","class balancing","boundary sampling","annotation efficiency"],"falsifier":"On a dataset with ground-truth boundaries, add synthetic high-uncertainty outlier pixels to background superpixels far from any boundary; if OREAL with max aggregation then prefers those superpixels and its mIoU drops below mean aggregation, the boundary-context mechanism is not what drives the gain. A simpler check is to measure the correlation between a superpixel's max pixel uncertainty and the density of true boundary pixels within it, since the argument predicts a positive correlation.","tokens_in":12690,"feed_emoji":"🎯","tokens_out":7725,"duration_ms":67182,"temperature":0.7,"pith_summary":"Active learning for semantic segmentation asks which image patches a human should label next. Existing patch-based methods score each superpixel, a group of visually similar pixels, by averaging per-pixel uncertainty, which dilutes the influence of hard boundary pixels. This paper argues that scoring by the maximum pixel uncertainty instead makes the method pick superpixels at object boundaries and provide useful context to the model. It adds a one-vs-rest entropy score that measures uncertainty about each class separately and, together with a per-class annotation budget, keeps the labeled set balanced. The claim is that this simple change improves segmentation quality across four datasets and three network architectures.","feed_headline":"Max pixel uncertainty beats mean for active learning","feed_subtitle":"Scoring superpixels by their most uncertain pixel samples object boundaries and improves semantic segmentation with fewer annotations.","key_machinery":"The mechanism has three parts. First, maximum aggregation: for a superpixel $K$ and pixel scores $s_i$, the superpixel score is $\\max_{i\\in K} s_i$, replacing the usual mean; this keeps boundary pixels from being averaged away. Second, one-vs-rest entropy: for a predicted class distribution $P_i$, the OVR entropy for class $c$ is $H_c[i] = -P_i[c]\\log P_i[c] - (1-P_i[c])\\log(1-P_i[c])$, and the superpixel score is $H_c[P] = \\max_{i\\in K} H_c[i]$. Third, class balancing: a vector $\\delta$ of required counts per class is computed by maximizing $\\min_c (n_c + \\delta_c)$ subject to $\\sum_c \\delta_c = Q$, where $n_c$ is the current number of labeled superpixels of class $c$ and $Q$ is the query size, and the $\\delta_c$ most uncertain superpixels for class $c$ are selected for annotation.","core_discovery":"The central claim is that the aggregation function used to turn pixel-level uncertainty into a superpixel-level score determines where a patch-based active learner looks. Average aggregation hides boundary pixels, so the paper's key move is maximum aggregation: the score of a superpixel is the maximum over its pixels of the per-pixel uncertainty. For multi-class segmentation the paper further defines one-vs-rest entropy per pixel as the binary entropy of $P_i[c]$ versus $1-P_i[c]$, then takes the maximum over the superpixel. A class-debt vector distributes the annotation budget across classes, so selecting the most OVR-uncertain superpixels for each class implicitly balances the labels. The paper reports that max aggregation alone improves mIoU by about 1.63 points on average across all active-learning iterations, and that OREAL reaches the highest area under the active-learning curve on the ResNet-101 and vision-transformer backbones.","pith_inferences":["Beyond the paper, the same extreme-pixel-rather-than-mean principle likely applies to other dense prediction tasks, such as depth estimation or panoptic segmentation, where boundary pixels are disproportionately informative.","One-vs-rest entropy could be adapted to image-level active learning by aggregating class-wise uncertainties over whole images, yielding a class-balancing acquisition function for full-image annotation.","A likely failure mode is that if a few pixels have high uncertainty from label noise or imaging artifacts rather than true boundaries, max aggregation will chase those uninformative superpixels; a robustness test on noisy superpixels would reveal this.","The class-debt formulation could be extended to non-uniform target label distributions or per-class annotation budgets, since the same optimization applies for any desired counts."],"forward_implications":["Any patch-based active-learning method for segmentation can be improved by replacing mean with maximum aggregation; the paper shows this across four baselines and three backbones.","One-vs-rest entropy combined with a per-class budget yields implicit class balancing without separate weighting heuristics.","Max aggregation selects superpixels adjacent to object boundaries, providing context that improves segmentation of both neighboring classes.","OREAL reaches a target accuracy, such as 95 percent of full-dataset performance on Pascal VOC, with fewer annotated superpixels when using max aggregation.","The improvements hold under both dominant-label and weak-label annotation schemes."],"supporting_citations":[{"why":"provides the superpixel dominant-labeling scheme, baseline, and optimal superpixel sizes.","marker":"[8]"},{"why":"supplies the weak-labeling annotation scheme and the PixelBal baseline.","marker":"[17]"},{"why":"contributes the class-count balancing idea and a baseline.","marker":"[7]"},{"why":"is the margin-based BvSB baseline.","marker":"[18]"},{"why":"supplies the training settings and the MONARCH dataset.","marker":"[35]"},{"why":"generates the SEEDS superpixels used as annotation units.","marker":"[14]"},{"why":"provides the DeepLabV3 architecture used as the segmentation model.","marker":"[10]"},{"why":"is the Cityscapes evaluation dataset.","marker":"[12]"},{"why":"is the Pascal VOC evaluation dataset.","marker":"[15]"},{"why":"is the EndoVis evaluation dataset.","marker":"[4]"}],"fun_headline_variants":["Max pixel uncertainty beats mean for active learning","One-vs-rest entropy and max pooling sharpen active learning","Boundary-aware active learning via max pixel uncertainty","OREL: max aggregation and OVR entropy for fewer labels","Use max, not mean, pixel uncertainty for active segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single most uncertain pixel in a superpixel tells you the superpixel is worth annotating; if that maximum is an outlier or label noise, the method will chase uninformative patches.","fun_headline_variants_meta":{"raw":{"variants":["Max pixel uncertainty beats mean for active learning","One-vs-rest entropy and max pooling sharpen active learning","Boundary-aware active learning via max pixel uncertainty","OREL: max aggregation and OVR entropy for fewer labels","Use max, not mean, pixel uncertainty for active segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3226,"prompt_tokens":845,"completion_tokens":2381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":2304}},"tokens_in":461,"tokens_out":2381,"duration_ms":18842,"temperature":1.0,"reasoning_tokens":2304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:36:13.120874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset with ground-truth boundaries, add synthetic high-uncertainty outlier pixels to background superpixels far from any boundary; if OREAL with max aggregation then prefers those superpixels and its mIoU drops below mean aggregation, the boundary-context mechanism is not what drives the gain. A simpler check is to measure the correlation between a superpixel's max pixel uncertainty and the density of true boundary pixels within it, since the argument predicts a positive correlation.","supporting_citations":[{"cited_title":"Revisiting superpixels for active learning in semantic seg- mentation with realistic annotation costs","cited_arxiv_id":null,"evidence_quote":"provides the superpixel dominant-labeling scheme, baseline, and optimal superpixel sizes."},{"cited_title":"Active learning for semantic segmentation with multi-class label query, 2023","cited_arxiv_id":null,"evidence_quote":"supplies the weak-labeling annotation scheme and the PixelBal baseline."},{"cited_title":"Class-balanced active learn- ing for image classification, 2021","cited_arxiv_id":null,"evidence_quote":"contributes the class-count balancing idea and a baseline."},{"cited_title":"Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos","cited_arxiv_id":null,"evidence_quote":"is the margin-based BvSB baseline."},{"cited_title":"Correlation-aware active learn- ing for surgery video segmentation, 2023","cited_arxiv_id":null,"evidence_quote":"supplies the training settings and the MONARCH dataset."},{"cited_title":"Seeds: Superpixels extracted via energy- driven sampling, 2013","cited_arxiv_id":null,"evidence_quote":"generates the SEEDS superpixels used as annotation units."},{"cited_title":"The cityscapes dataset for semantic urban scene understanding","cited_arxiv_id":null,"evidence_quote":"is the Cityscapes evaluation dataset."},{"cited_title":"Everingham, L","cited_arxiv_id":null,"evidence_quote":"is the Pascal VOC evaluation dataset."}],"review_version":1}