REVIEW 4 major objections 5 minor 39 references
Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that replacing mean with maximum aggregation of pixel-wise uncertainty in superpixel scoring improves boundary sampling and segmentation quality, and that a one-vs-rest entropy score adds implicit class balancing.
desk verdict Max aggregation is a solid, simple win; the one-vs-rest entropy claim is not isolated and the class-balancing mechanism has a hole. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism has three parts. First, maximum aggregation: for a superpixel $K$ and pixel scores $s_i$, the superpixel score is $\max_{i\in K} s_i$, replacing the usual mean; this keeps boundary pixels from being averaged away. Second, one-vs-rest entropy: for a predicted class distribution $P_i$, the OVR entropy for class $c$ is $H_c[i] = -P_i[c]\log P_i[c] - (1-P_i[c])\log(1-P_i[c])$, and the superpixel score is $H_c[P] = \max_{i\in K} H_c[i]$. Third, class balancing: a vector $\delta$ of required counts per class is computed by maximizing $\min_c (n_c + \delta_c)$ subject to $\sum_c \delta_c = Q$, where $n_c$ is the current number of labeled superpixels of class $c$ and $Q$ is the query size, and the $\delta_c$ most uncertain superpixels for class $c$ are selected for annotation.
What would settle it
On a dataset with ground-truth boundaries, add synthetic high-uncertainty outlier pixels to background superpixels far from any boundary; if OREAL with max aggregation then prefers those superpixels and its mIoU drops below mean aggregation, the boundary-context mechanism is not what drives the gain. A simpler check is to measure the correlation between a superpixel's max pixel uncertainty and the density of true boundary pixels within it, since the argument predicts a positive correlation.
Extended reading notes
Core claim
The central claim is that the aggregation function used to turn pixel-level uncertainty into a superpixel-level score determines where a patch-based active learner looks. Average aggregation hides boundary pixels, so the paper's key move is maximum aggregation: the score of a superpixel is the maximum over its pixels of the per-pixel uncertainty. For multi-class segmentation the paper further defines one-vs-rest entropy per pixel as the binary entropy of $P_i[c]$ versus $1-P_i[c]$, then takes the maximum over the superpixel. A class-debt vector distributes the annotation budget across classes, so selecting the most OVR-uncertain superpixels for each class implicitly balances the labels. The paper reports that max aggregation alone improves mIoU by about 1.63 points on average across all active-learning iterations, and that OREAL reaches the highest area under the active-learning curve on the ResNet-101 and vision-transformer backbones.
Load-bearing premise
The load-bearing premise is that the single most uncertain pixel in a superpixel tells you the superpixel is worth annotating; if that maximum is an outlier or label noise, the method will chase uninformative patches.
Editorial extensions
If this is right
- Any patch-based active-learning method for segmentation can be improved by replacing mean with maximum aggregation; the paper shows this across four baselines and three backbones.
- One-vs-rest entropy combined with a per-class budget yields implicit class balancing without separate weighting heuristics.
- Max aggregation selects superpixels adjacent to object boundaries, providing context that improves segmentation of both neighboring classes.
- OREAL reaches a target accuracy, such as 95 percent of full-dataset performance on Pascal VOC, with fewer annotated superpixels when using max aggregation.
- The improvements hold under both dominant-label and weak-label annotation schemes.
Reading between the lines
- Beyond the paper, the same extreme-pixel-rather-than-mean principle likely applies to other dense prediction tasks, such as depth estimation or panoptic segmentation, where boundary pixels are disproportionately informative.
- One-vs-rest entropy could be adapted to image-level active learning by aggregating class-wise uncertainties over whole images, yielding a class-balancing acquisition function for full-image annotation.
- A likely failure mode is that if a few pixels have high uncertainty from label noise or imaging artifacts rather than true boundaries, max aggregation will chase those uninformative superpixels; a robustness test on noisy superpixels would reveal this.
- The class-debt formulation could be extended to non-uniform target label distributions or per-class annotation budgets, since the same optimization applies for any desired counts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OREAL, a patch-based active learning method for semantic segmentation. It makes two main contributions: (i) replacing mean aggregation of pixel-wise uncertainty scores with maximum aggregation when scoring superpixels, in order to sample boundary regions, and (ii) introducing one-vs-rest (OVR) entropy, a per-class uncertainty score intended to implicitly balance the class distribution of the selected annotation set. The method is evaluated on four datasets (Cityscapes, Pascal VOC, MONARCH, EndoVis) with three backbones (ResNet-50, ResNet-101, ViT), against several baselines, using the AuALC metric over six active learning steps. The experiments show that on average, maximum aggregation improves results over mean aggregation, and that OREAL is competitive with or better than baselines on most datasets, except for MONARCH, where OREAL is generally outperformed. The paper also presents an extension to weak labeling and an analysis of annotation cost to reach 95% of full-training performance.
Significance. If the claims hold, the paper would make a useful practical contribution to active learning for semantic segmentation. The maximum-aggregation idea is simple, easy to implement, and appears to yield consistent average gains across diverse datasets and architectures, as evidenced by the paired mean/max comparisons in Table 1. The OVR entropy score is a novel formulation, and the paper includes a thorough experimental protocol with multiple runs and several datasets. The explicit acknowledgement of the MONARCH counterexample and the analysis of superpixel dominant-label noise are also strengths. However, the significance is tempered by two gaps: the OVR component is never isolated experimentally, so the paper's central novelty beyond maximum aggregation is not directly supported, and the claimed implicit class balancing rests on an unverified assumption about the reliability of per-class uncertainty selection. These issues are fixable with additional experiments and analysis, and the paper's core hypothesis about maximum aggregation is largely supported by the data.
major comments (4)
- [Sec. 3.3/3.5, Algorithm 1, Table 1, Fig. 4] The contribution of the one-vs-rest entropy is never isolated. Table 1 compares full OREAL against baselines, and Fig. 4 ablates mean vs. max aggregation for all methods, but there is no variant of OREAL that uses maximum aggregation with a standard entropy (or another per-class score) under the same per-class quota δc. Without such an ablation, the observed gains cannot be attributed to OVR entropy rather than to the max aggregation or to the explicit quota mechanism in Algorithm 1. Since the class quota is already enforced by steps 6–10 of Algorithm 1, the per-class balancing is achieved by the quota itself; OVR entropy only selects within each quota. The paper should either provide a direct comparison of OVR entropy with a non-OVR score under identical quota conditions, or substantially soften the claim that the OVR score provides a distinct benefit.
- [Sec. 3.5, Algorithm 1] The statement that 'any deviation from δc is compensated for in subsequent iterations' is unsupported. Algorithm 1 selects the top-δc superpixels by predicted class c, but the true dominant label of a selected superpixel may differ from c, as the paper itself acknowledges. The claimed compensation is deferred to an Appendix that is not included in the manuscript. The paper should provide empirical evidence on the fraction of selected superpixels whose true label matches the requested class, and demonstrate that the actual labeled class distribution converges to the balanced target over iterations. Without this, the 'implicit class balancing' claim is not established.
- [Table 1, Fig. 4, Sec. 4.3] The experimental support for the central claim is weakened by the absence of error bars or significance tests in Table 1, despite the paper reporting multiple runs. Fig. 4 shows error bars, but Table 1 is used for the main cross-method comparisons. Moreover, the MONARCH dataset systematically contradicts the general claim: for example, OREAL max achieves 69.7 AuALC on MONARCH with ResNet-50, while Revisiting SP max achieves 73.7, and similar gaps appear with ResNet-101. The paper acknowledges this but does not analyze why OREAL fails on MONARCH. Given the paper's claim of comprehensive validation, the authors should provide a statistical comparison (e.g., paired tests with variance information) and a more detailed investigation of when max aggregation and OVR entropy help or hurt, including the MONARCH setting.
- [Sec. 3.2, Fig. 2] The load-bearing assumption that the maximum pixel-wise uncertainty is a reliable proxy for a superpixel's annotation value is not examined. A single noisy or outlier pixel can dominate the max score, and the paper provides no analysis of how often this occurs or how sensitive the results are to this choice. A robustness study (e.g., comparing max with other extreme quantiles, or analyzing the label purity of selected superpixels) would strengthen the central claim. This is not a fatal flaw, but it is a significant missing analysis for a method whose entire rationale rests on the max operation.
minor comments (5)
- [Throughout] There are several typos and grammatical issues, for example 'we found out that using' in Sec. 3.2, 'A verage' in Table 1, and 'a novel uncertainty score function' in the Abstract. A careful proofreading pass is recommended.
- [Fig. 4 caption] The caption states that error bars indicate one standard deviation, but Table 1, which is the primary quantitative summary, does not report any variance information. Please make the reporting consistent.
- [Sec. 4.2] The sentence 'This labeling method is known as the dominant label' appears in Sec. 2.2 rather than Sec. 4.2; the notation for the split proportions is also a bit cumbersome (e.g., '4 : 1for EndoVis'). Consider reformatting for clarity.
- [Sec. 5] In the superpixel dominant labeling discussion, the sentence 'A model A is trained... and a model B is trained...' lacks the verb 'is' before 'trained' in the second clause; this is likely a typo.
- [References] The paper repeatedly refers to an Appendix for details on the compensation mechanism and for additional results, but the submitted manuscript does not include the appendix. Please either include the appendix or remove the references to it.
Circularity Check
No significant circularity: OREAL's components are empirically evaluated, not derived from their own outputs; the acknowledged class-balance deviation is a limitation, not a circular construction.
full rationale
The paper does not derive any prediction from a fitted parameter or from a self-citation chain. Maximum aggregation and one-vs-rest entropy are proposed as design choices and validated on held-out test sets against external baselines (random, BvSB, revisiting SP, PixelBal, CBAL). The only self-citation is [35], used for training settings and the MONARCH dataset; it is not load-bearing for the central claim. The acknowledged issue that OVR entropy may select superpixels whose true label is not class c (Sec. 3.5) is an unsupported empirical assumption about subsequent-iteration compensation, not a circular reduction: the method can fail without making the result equal to its input. The lack of an ablation isolating OVR from max aggregation weakens causal attribution but is an experimental design gap, not circularity. The evaluation is self-contained against external benchmarks, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- AL budget multiplier Q =
50 x P (20 x P for MONARCH)
- Superpixel count per image =
196 (EndoVis), 322 (Pascal VOC), 36 (MONARCH), 576 (Cityscapes), 256 (Pascal VOC)
assumptions (3)
- domain assumption Superpixels preserve semantic boundaries well enough that dominant labeling yields useful training labels.
- domain assumption The segmentation model's softmax predictions provide meaningful uncertainty estimates for the active learning scores.
- domain assumption The training protocol from [35] is suitable for all four datasets and all three backbones.
Cite this review
Pith. "Pith review of Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation." pith.science (2026). https://pith.science/paper/YH63FLXO
@misc{pith2026241206470,
author = {Pith},
title = {Pith review of: Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YH63FLXO}},
note = {Machine review of arXiv:2412.06470}
}
read the original abstract
Multi-class semantic segmentation remains a cornerstone challenge in computer vision. Yet, dataset creation remains excessively demanding in time and effort, especially for specialized domains. Active Learning (AL) mitigates this challenge by selecting data points for annotation strategically. However, existing patch-based AL methods often overlook boundary pixels critical information, essential for accurate segmentation. We present OREAL, a novel patch-based AL method designed for multi-class semantic segmentation. OREAL enhances boundary detection by employing maximum aggregation of pixel-wise uncertainty scores. Additionally, we introduce one-vs-rest entropy, a novel uncertainty score function that computes class-wise uncertainties while achieving implicit class balancing during dataset creation. Comprehensive experiments across diverse datasets and model architectures validate our hypothesis.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/vainf/deeplabv3plus-pytorch. 5
-
[2]
Ac- tive learning for imbalanced datasets, 2020
Umang Aggarwal, Adrian Popescu, and C ´eline Hudelot. Ac- tive learning for imbalanced datasets, 2020. W ACV . 2, 3
work page 2020
-
[3]
Minority Class Oriented Active Learning for Imbalanced Datasets
Umang Aggarwal, Adrian Popescu, and C ´eline Hudelot. Mi- nority class oriented active learning for imbalanced datasets. CoRR, abs/2202.00390, 2022. 2, 3
work page Pith review arXiv 2022
-
[4]
2018 robotic scene segmentation chal- lenge
Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Ste- fan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, F´elix Fuentes-Hurtado, Evangello Flouty, Ahmed Kedir Mo- hammed, Marius Pedersen, Avinash Kori, Alex Varghese, Ganapathy Krishnamurthi, David Rauber, Robert Mendel, Christoph Palm, Sophia Bano, Guinther Saibro, Chi-Sheng Shih, Hsun-An Chiang, Juntang Z...
arXiv 2018
-
[5]
Ricardo Barata, Miguel Leite, Ricardo Pacheco, Marco O. P. Sampaio, Jo ˜ao Tiago Ascens ˜ao, and Pedro Bizarro. Active learning for online training in imbalanced data streams under cold start. CoRR, abs/2107.07724, 2021. 2, 3
work page Pith review arXiv 2021
-
[6]
Active Class Incremental Learning for Imbalanced Datasets
Eden Belouadah, Adrian Popescu, Umang Aggarwal, and L´eo Saci. Active class incremental learning for imbalanced datasets. CoRR, abs/2008.10968, 2020. 2, 3
work page Pith review arXiv 2008
-
[7]
Class-balanced active learn- ing for image classification, 2021
Javad Zolfaghari Bengar, Joost van de Weijer, Laura Lopez Fuentes, and Bogdan Raducanu. Class-balanced active learn- ing for image classification, 2021. 2, 3, 5, 6, 8
work page 2021
-
[8]
Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo. Revisiting superpixels for active learning in semantic seg- mentation with realistic annotation costs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10988–10997, June 2021. 1, 2, 3, 5, 6, 7, 8
work page 2021
Show all 39 references
-
[9]
Pinheiro, Negar Rostamzadeh, and Christopher J
Arantxa Casanova, Pedro O. Pinheiro, Negar Rostamzadeh, and Christopher J. Pal. Reinforced active learning for image segmentation. CoRR, abs/2002.06583, 2020. 1, 2
2002 arXiv
-
[10]
Rethinking atrous convolution for semantic image segmentation
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017. 5
2017 arXiv
-
[11]
Metabox+: A new region based ac- tive learning method for semantic segmentation using pri- ority maps
Pascal Colling, Lutz Roese-Koerner, Hanno Gottschalk, and Matthias Rottmann. Metabox+: A new region based ac- tive learning method for semantic segmentation using pri- ority maps. CoRR, abs/2010.01884, 2020. 1, 2
2010 arXiv
-
[12]
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognitio...
2016
-
[13]
An- gelini, Yike Guo, and Wenjia Bai
Chengliang Dai, Shuo Wang, Yuanhan Mo, Elsa D. An- gelini, Yike Guo, and Wenjia Bai. Suggestive annotation of brain tumour images with gradient-guided sampling. CoRR, abs/2006.14984, 2020. 1, 2
2006 arXiv
-
[14]
Seeds: Superpixels extracted via energy- driven sampling, 2013
Michael Van den Bergh, Xavier Boix, Gemma Roig, and Luc Van Gool. Seeds: Superpixels extracted via energy- driven sampling, 2013. 2, 5
2013
-
[15]
Everingham, L
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) chal- lenge. International Journal of Computer Vision, 88(2):303– 338, June 2010. 5, 6, 8
2010
-
[16]
Alireza Golestaneh and Kris M
S. Alireza Golestaneh and Kris M. Kitani. Importance of self-consistency in active learning for semantic segmenta- tion. CoRR, abs/2008.01860, 2020. 1, 2
2008 arXiv
-
[17]
Active learning for semantic segmentation with multi-class label query, 2023
Sehyun Hwang, Sohyun Lee, Hoyoung Kim, Minhyeon Oh, Jungseul Ok, and Suha Kwak. Active learning for semantic segmentation with multi-class label query, 2023. 1, 2, 3, 5, 6, 7, 8
2023
-
[18]
Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos
Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 2372–2379, 2009. 5, 6, 8
2009
-
[19]
Hegde, V
Tejaswi Kasarla, G Nagendar, Guruprasad M. Hegde, V . Bal- asubramanian, and C.V . Jawahar. Region-based active learn- ing for efficient labeling in semantic segmentation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1109–1117, 2019. 1, 2
2019
-
[20]
Weakly supervised semantic labelling and instance segmentation
Anna Khoreva, Rodrigo Benenson, Jan Hendrik Hosang, Matthias Hein, and Bernt Schiele. Weakly supervised semantic labelling and instance segmentation. CoRR, abs/1603.07485, 2016. 2
2016 arXiv
-
[21]
Adaptive superpixel for active learning in semantic segmentation, 2023
Hoyoung Kim, Minhyeon Oh, Sehyun Hwang, Suha Kwak, and Jungseul Ok. Adaptive superpixel for active learning in semantic segmentation, 2023. 1, 2
2023
-
[22]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. 7
2014
-
[23]
Clinical: Targeted active learning for imbalanced medical image clas- sification, 2022
Suraj Kothawade, Atharv Savarkar, Venkat Iyer, Lakshman Tamil, Ganesh Ramakrishnan, and Rishabh Iyer. Clinical: Targeted active learning for imbalanced medical image clas- sification, 2022. 2, 3
2022
-
[24]
Weakly supervised segmentation of small buildings with point labels
Jae-Hun Lee, ChanYoung Kim, and Sanghoon Sull. Weakly supervised segmentation of small buildings with point labels. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7386–7395, 2021. 2
2021
-
[25]
DIAL: deep interactive and active learning for semantic segmentation in remote sensing
Gaston Lenczner, Adrien Chan-Hon-Tong, Bertrand Le Saux, Nicola Luminari, and Guy Le Besnerais. DIAL: deep interactive and active learning for semantic segmentation in remote sensing. CoRR, abs/2201.01047, 2022. 1, 2
2022 arXiv
-
[26]
CEREALS - cost- effective region-based active learning for semantic segmen- tation
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. CEREALS - cost- effective region-based active learning for semantic segmen- tation. CoRR, abs/1810.09726, 2018. 1, 2
2018 arXiv
-
[27]
Weakly-supervised se- mantic segmentation by learning label uncertainty
Robby Neven, Davy Neven, Bert De Brabandere, Marc Proesmans, and Toon Goedem ´e. Weakly-supervised se- mantic segmentation by learning label uncertainty. CoRR, abs/2110.05926, 2021. 2
2021 arXiv
-
[28]
Bly, Kris S
Haonan Peng, Shan Lin, Daniel King, Yun-Hsuan Su, Ran- dall A. Bly, Kris S. Moe, and Blake Hannaford. Reducing annotating load: Active learning with synthetic images in surgical instrument segmentation. CoRR, abs/2108.03534,
-
[29]
Bearman, Vittorio Ferrari, and Li Fei-Fei
Olga Russakovsky, Amy L. Bearman, Vittorio Ferrari, and Li Fei-Fei. What’s the point: Semantic segmentation with point supervision. CoRR, abs/1506.02106, 2015. 2
2015 arXiv
-
[30]
A mathematical theory of commu- nication
Claude Elwood Shannon. A mathematical theory of commu- nication. The Bell System Technical Journal , 27:379–423,
-
[31]
Viewal: Active learning with viewpoint entropy for semantic segmentation
Yawar Siddiqui, Julien Valentin, and Matthias Nießner. Viewal: Active learning with viewpoint entropy for semantic segmentation. CoRR, abs/1911.11789, 2019. 1, 2
1911 arXiv
-
[32]
Vari- ational adversarial active learning
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Vari- ational adversarial active learning. CoRR, abs/1904.00370,
1904 arXiv
-
[33]
MEAL: manifold embedding-based active learning
Deepthi Sreenivasaiah and Thomas Wollmann. MEAL: manifold embedding-based active learning. CoRR, abs/2106.11858, 2021. 1, 2
2021 arXiv
-
[34]
Segmenter: Transformer for semantic seg- mentation
Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic seg- mentation. CoRR, abs/2105.05633, 2021. 5
2021 arXiv
-
[35]
Correlation-aware active learn- ing for surgery video segmentation, 2023
Fei Wu, Pablo Marquez-Neila, Mingyi Zheng, Hedyeh Rafii- Tari, and Raphael Sznitman. Correlation-aware active learn- ing for surgery video segmentation, 2023. 1, 2, 5, 6, 7
2023
-
[36]
Towards fewer annotations: Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation, 2022
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, and Xinjing Cheng. Towards fewer annotations: Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation, 2022. 1, 2, 3
2022
-
[37]
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z. Chen. Suggestive annotation: A deep active learn- ing framework for biomedical image segmentation. CoRR, abs/1706.04737, 2017. 1, 2
2017 arXiv
-
[38]
Algorithm selection for deep active learning with imbalanced datasets, 2023
Jifan Zhang, Shuai Shao, Saurabh Verma, and Robert Nowak. Algorithm selection for deep active learning with imbalanced datasets, 2023. 2, 3
2023
-
[39]
Dsal: Deeply supervised active learning from strong and weak labelers for biomedical image segmentation
Ziyuan Zhao, Zeng Zeng, Kaixin Xu, Cen Chen, and Cuntai Guan. Dsal: Deeply supervised active learning from strong and weak labelers for biomedical image segmentation. IEEE Journal of Biomedical and Health Informatics, 25(10):3744– 3751, 2021. 1, 2
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.