REVIEW 3 major objections 6 minor 1 cited by
Adaptive Prototype Replay for Class Incremental Semantic Segmentation
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Adaptive prototype replay for class-incremental semantic segmentation shifts stored class memories with the feature drift, and the paper reports it improves on both Pascal VOC and ADE20K.
desk verdict A genuine training-free prototype-compensation idea for CISS, but the paper hasn't shown that the adaptive mechanism is actually what drives its headline multi-step gains; major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
At the core is the adaptive deviation compensation (ADC) strategy: a representation-shift vector $\Delta_{c}^{t-1\to t} = \Omega(\hat P_c^{t-1}, \hat P_c^t)$ computed as the difference between sub-prototypes extracted by the previous and current feature extractors on the same old-class pixels, selected by unified high-confidence prediction masks from both models. The shifted prototype is blended with the stored prototype using an adaptive weight $\rho$ proportional to the number of confident co-occurring pixels, so replay degrades gracefully when the estimate is weak. Two loss terms carry the rest of the argument: the uncertainty-aware constraint (UAC) computes certainty as the gap between the top two sigmoid logits and penalizes uncertain pixels, compressing each class's features into a compact region; the compensation-based prototype discrimination (CPD) loss repels new-class feature centers, misclassified-background centers, and the compensated old-class prototypes from one another. Together the machinery turns the stored memory into a movable object that tracks the model's current representation.
What would settle it
Measure Adapter against a hard ablation that forces the adaptive weight $\rho$ to zero, making replay purely fixed while keeping every other loss, across VOC 2-2 and 1-1; if full ADC does not beat that ablated version on old-class mIoU, the drift estimate is not carrying the reported gains. Alternatively, build a controlled CISS split in which no incremental training image contains any old-class pixel, so the co-occurrence requirement is violated; the ADC component should then add nothing over fixed replay, and if it still does, another mechanism is responsible.
Extended reading notes
Core claim
The central claim is that catastrophic forgetting in prototype-replay CISS comes not only from classifier bias but from a representation gap: between the moment a prototype is saved and the moment it is replayed, the network's feature extractor has been updated on new data, so the saved prototype describes the old model's representation, not the current one. Adapter removes that gap with a training-free estimate. For each old class, it takes pixels in current-step images that the old and current models both predict as that class with high certainty, computes a sub-prototype with each feature extractor on those identical pixels, and uses the displacement vector between the two sub-prototypes as the representation shift. The stored prototype is updated to $\hat P_c^{t-1} = P_c^{t-1} + \Delta_{c}^{t-1\to t}$, then blended with the original prototype by an adaptive weight that depends on how many confident co-occurring pixels were available. This compensatory prototype is the memory that gets replayed. The paper's empirical claim is that this drift-corrected replay, together with the UAC and CPD losses, is the reason for the reported gains over prior methods.
Load-bearing premise
The load-bearing premise is that enough old-class pixels appear in the current step's training images, and are confidently and consistently predicted by both the previous and current models, to give a reliable estimate of each old class's representation shift; when that fails, the method falls back on the original fixed prototype.
Editorial extensions
If this is right
- Correcting representation drift at every step prevents small per-step errors from accumulating, which matches the reported gains in the 2-2 (10 steps) and 1-1 (20 steps) settings.
- No old images need to be stored for the drift correction; the estimate comes from old-class pixels that co-occur with new training data, preserving the storage and privacy benefits of prototype replay.
- If the uncertainty-aware constraint is the main carrier of the gains, it should transfer to other CISS methods; the paper reports consistent improvements when UAC is added to several prior methods.
- The mechanism is backbone-independent in the reported experiments: integrating with a Swin-B transformer-based segmentation model gives gains on VOC 15-1, and adding a tiny 50-sample exemplar memory improves results further.
- The approach remains compatible with memory-based replay, so drift compensation and stored exemplars can be combined rather than treated as alternatives.
Reading between the lines
- An untested consequence is that ADC's gain should scale with the amount of old-class pixels available in incremental batches; in strictly disjoint splits or datasets where old and new categories rarely co-occur, the method should converge toward ordinary fixed-prototype replay, predicting smaller but still positive improvements.
- The same drift-compensation scheme could be dropped into other replay-based continual learners that keep prototypes or feature statistics and update a shared backbone, such as incremental object detection or instance segmentation; the only requirement is a way to obtain the same old-class identities from two consecutive models.
- A sharper ablation prediction is that UAC's benefit should correlate with class-wise intra-class feature variance in the current model; measuring that variance before and after applying UAC would test the stated compactness mechanism directly.
- The certainty score defined as the gap between the top two sigmoid logits is a cheap pseudo-label quality filter, so applying it to background-shift pseudo-labeling outside the prototype-replay context is a direct extension the paper does not pursue.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Adapter, a prototype-replay method for class-incremental semantic segmentation (CISS). Its central claim is that existing prototype replay fails because stored prototypes are fixed while the feature representation drifts as incremental steps update shared weights. To address this, Adapter introduces an adaptive deviation compensation (ADC) strategy that estimates each old class's representation shift from old-class pixels co-occurring in the current training set and uses the estimate to update stored prototypes (Eqs. 2-8). Two auxiliary losses are added: an uncertainty-aware constraint (UAC) loss to compact class features (Eqs. 9-11) and a compensation-based prototype discrimination (CPD) loss to separate new-class features from old-class prototypes (Eqs. 12-16). Experiments on Pascal VOC and ADE20K report state-of-the-art mIoU, with the largest gains in long multi-step scenarios such as VOC 2-2 and 1-1, and ablations on VOC 15-1 support the contribution of each component.
Significance. If substantiated, the paper makes a useful contribution by identifying a real limitation of fixed-prototype replay and proposing a training-free mechanism (ADC) to compensate representation drift. The UAC loss appears portable: Table 5 reports consistent gains when applied to MiB, PLOP, DKD, and CoinSeg, which strengthens the evidence for the underlying uncertainty-compaction idea. The manuscript also includes a memory-based variant, hyperparameter sensitivity analysis, and qualitative results, and the code is released. However, the current evidence is not yet sufficient for the headline multi-step claim: the component ablation is only on 15-1, and no statistics are reported on how often the ADC mechanism is actually active in long-step settings. In addition, the UAC loss as written depends on an undefined target term, preventing reproducibility of a component that is credited with part of the reported gains.
major comments (3)
- [Method, Eq. (11)] The UAC loss is not fully specified: Eq. (11) defines L_uac = d(u_t^i ⊙ m_t^i, Ω_t^i), but neither the 'ideal targets' Ω_t^i nor the distance function d(·) is ever defined. The text only says d denotes a distance measurement function. Since Table 3 attributes a 0.5 mIoU improvement in all classes and 1.3 in new classes to UAC, and Table 5 uses UAC as a plug-in module, this missing definition is load-bearing for reproducibility and for the component claim. Please define Ω_t^i and d explicitly, or state the concrete instantiation used in the experiments.
- [Adaptive Deviation Compensation, Eq. (8); Tables 1 and 3] The paper's headline claim of superiority 'particularly in challenging multi-step scenarios' is based on Table 1, where Adapter outperforms STAR by 3.0 mIoU on 2-2 and 6.2 mIoU on 1-1. However, the only component ablation (Table 3) is on 15-1, a setting where the initial step contains many classes and co-occurrence of old-class pixels with the new class is relatively abundant. In the 1-1 setting, each incremental step introduces only one new class, and the numerator of ρ_c in Eq. (8) (the count of unified-mask pixels for an old class in the current sub-dataset) may be very small or zero for many old classes, making ρ_c near zero and reducing the compensatory prototype in Eq. (7) to the unshifted stored prototype. The paper reports no statistics on these pixel counts or on the resulting ρ_c values, and no ablation on 2-2 or 1-1. Consequently, the large multi-step gains cannot currently be attributed to the ADC mechanism rather than to the UAC/CPD losses or other training choices. Please add a multi-step ablation and report the coverage/frequency of nonzero ρ_c values.
- [Experiments, Tables 1 and 2] Several of the reported gains over the prior state of the art are small: on ADE20K, the all-class mIoU improvements over STAR are 0.3, 0.3, 0.4, and 0.8 for 100-50, 50-50, 100-10, and 100-5, respectively; on VOC, the 15-1 and 10-1 gains are 0.7 and 0.4. No variance estimates or repeated-run statistics are provided, so it is unclear whether these differences are within run-to-run noise. Because the abstract and Section 'Experimental Results' claim state-of-the-art results, please report standard deviations over at least three seeds or explicitly justify why the comparison protocol does not require them.
minor comments (6)
- [Abstract] There is a typographical error: 'adaptive deviation compen sation' should be 'adaptive deviation compensation'.
- [Method, Eq. (13)] In Eq. (13), the minimization uses the same symbol c in the outer sum over new classes and in the inner minimum over old classes; please use a different index (e.g., c') for the old-class prototype to avoid confusion and to make clear that the min is over old-class prototypes.
- [Method, Eqs. (5) and (11)] The symbol Ω is overloaded: Eq. (5) uses Ω(·,·) for the displacement-vector operation, while Eq. (11) uses Ω_t^i for the ideal uncertainty targets. Please use distinct symbols for these two quantities.
- [Experiments, Section 'Comparison on Pascal VOC'] The text says '1-1 (steps)' but the setting is 1-1 (20 steps); the step count is missing.
- [Table 4 caption] The abbreviation 'Copro' is used without definition; please spell out that it denotes using the compensated old-class prototypes (via ADC) in the L_n↔o term.
- [Related work / References] The reference to Representation Compensation Networks (Zhang et al. 2022a) appears in the reference list but is neither discussed in the related-work section nor included in the experimental comparisons; please either add a brief discussion and comparison or remove the reference.
Circularity Check
No circularity found: Adapter's ADC is an empirical heuristic, losses come from external baselines, and the reported gaps are evidence-strength issues rather than definitional tautologies.
full rationale
Adapter is an empirical pipeline rather than a derivation. The central ADC mechanism (Eqs. 5-8) estimates a displacement vector from old- and current-model sub-prototypes on co-occurring old-class pixels and uses it to form a compensatory prototype; this is a heuristic estimator, not a quantity fitted to the evaluation target and then renamed as a prediction. The replayed-distribution losses L_mbce and L_kd are explicitly taken from external works (Cha et al. 2021; Chen et al. 2023), whose authors do not overlap with the present paper, so there is no load-bearing self-citation chain. No uniqueness theorem or prior-work ansatz is imported to force the method's choice. The UAC and CPD losses are new components with ablations (Tables 3-5), and the fact that hyperparameters are tuned on the same validation sets or that the ADC ablation is only on 15-1 is a limitation in evidence strength, not a definitional circularity: the reported gains are not logically entailed by the loss definitions. The skeptical concern that rho_c may be near zero in 1-1 and 2-2 settings is an empirical support gap, not a circular step.
Assumptions & free parameters
free parameters (4)
- alpha (KD weight) =
5
- beta (UAC weight) =
0.1
- gamma (CPD weight) =
0.05
- tau (confidence threshold) =
0.7
assumptions (3)
- domain assumption Old-class pixels are present in the current incremental training dataset and are confidently predicted by both the previous and current models, so that their features can be used to estimate representation shift.
- domain assumption Replaying Gaussian-distributed features around stored prototypes approximates the true old-class feature distribution well enough to mitigate classifier bias.
- domain assumption The unified prediction mask in Eq. 3 aligns old and current model predictions on the same spatial regions, yielding comparable sub-prototypes.
Cite this review
Pith. "Pith review of Adaptive Prototype Replay for Class Incremental Semantic Segmentation." pith.science (2026). https://pith.science/paper/KN75TGOM
@misc{pith2026241212669,
author = {Pith},
title = {Pith review of: Adaptive Prototype Replay for Class Incremental Semantic Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/KN75TGOM}},
note = {Machine review of arXiv:2412.12669}
}
read the original abstract
Class incremental semantic segmentation (CISS) aims to segment new classes during continual steps while preventing the forgetting of old knowledge. Existing methods alleviate catastrophic forgetting by replaying distributions of previously learned classes using stored prototypes or features. However, they overlook a critical issue: in CISS, the representation of class knowledge is updated continuously through incremental learning, whereas prototype replay methods maintain fixed prototypes. This mismatch between updated representation and fixed prototypes limits the effectiveness of the prototype replay strategy. To address this issue, we propose the Adaptive prototype replay (Adapter) for CISS in this paper. Adapter comprises an adaptive deviation compen sation (ADC) strategy and an uncertainty-aware constraint (UAC) loss. Specifically, the ADC strategy dynamically updates the stored prototypes based on the estimated representation shift distance to match the updated representation of old class. The UAC loss reduces prediction uncertainty, aggregating discriminative features to aid in generating compact prototypes. Additionally, we introduce a compensation-based prototype similarity discriminative (CPD) loss to ensure adequate differentiation between similar prototypes, thereby enhancing the efficiency of the adaptive prototype replay strategy. Extensive experiments on Pascal VOC and ADE20K datasets demonstrate that Adapter achieves state-of-the-art results and proves effective across various CISS tasks, particularly in challenging multi-step scenarios. The code and model is available at https://github.com/zhu-gl-ux/Adapter.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
IPSeg: Image Posterior Mitigates Semantic Drift in Class-Incremental Segmentation
IPSeg reduces semantic drift in incremental segmentation by multiplying pixel predictions with image posterior probabilities and decoupling permanent background semantics from temporary foreground semantics.
Reference graph
Works this paper leans on
-
[1]
Baek, D.; Oh, Y.; Lee, S.; Lee, J.; and Ham, B. 2022. Decomposed knowledge distillation for class-incremental semantic segmentation. Advances in Neural Information Processing Systems, 35
work page 2022
-
[2]
Cermelli, F.; Mancini, M.; Bulo, S. R.; Ricci, E.; and Caputo, B. 2020. Modeling the background for incremental learning in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9233--9242
work page 2020
-
[3]
Cha, S.; Yoo, Y.; Moon, T.; et al. 2021. Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. Advances in neural information processing systems, 34
work page 2021
-
[4]
Chen, J.; Cong, R.; Luo, Y.; Ip, H.; and Kwong, S. 2023. Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation. Advances in Neural Information Processing Systems, 36
work page 2023
-
[5]
Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834--848
2017
-
[6]
Douillard, A.; Chen, Y.; Dapogny, A.; and Cord, M. 2021. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4040--4050
work page 2021
-
[7]
Douillard, A.; Cord, M.; Ollion, C.; Robert, T.; and Valle, E. 2020. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Proceedings of the European conference on computer vision (ECCV), 86--102. Springer
work page 2020
-
[8]
K.; Winn, J.; and Zisserman, A
Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303--338
2010
Show all 36 references
-
[9]
J.; and Tao, D
Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6): 1789--1819
2021
-
[10]
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778
2016
-
[11]
Jung, S.; Ahn, H.; Cha, S.; and Moon, T. 2020. Continual learning with node-importance based adaptive group sparse regularization. Advances in neural information processing systems, 33
2020
-
[12]
A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al
Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017. Overcoming catastrophic forgetting in neural networks. volume 114, 3521--3526. National Acad Sciences
2017
-
[13]
Li, Z.; and Hoiem, D. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12): 2935--2947
2017
-
[14]
Lopez-Paz, D.; and Ranzato, M. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30
2017
-
[15]
Maracani, A.; Michieli, U.; Toldo, M.; and Zanuttigh, P. 2021. Recall: Replay-based continual learning in semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, 7026--7035
2021
-
[16]
D.; and Van De Weijer, J
Masana, M.; Liu, X.; Twardowski, B.; Menta, M.; Bagdanov, A. D.; and Van De Weijer, J. 2022. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5): 5513--5533
2022
-
[17]
McCloskey, M.; and Cohen, N. J. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, 109--165. Elsevier
1989
-
[18]
Michieli, U.; and Zanuttigh, P. 2019. Incremental learning techniques for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision workshops
2019
-
[19]
Michieli, U.; and Zanuttigh, P. 2021. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1114--1124
2021
-
[20]
Oh, Y.; Baek, D.; and Ham, B. 2022. Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. Advances in Neural Information Processing Systems, 35
2022
-
[21]
H.; Phung, S
Phan, M. H.; Phung, S. L.; Tran-Thanh, L.; Bouzerdoum, A.; et al. 2022. Class similarity weighted knowledge distillation for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16866--16875
2022
-
[22]
Rebuffi, S.-A.; Kolesnikov, A.; Sperl, G.; and Lampert, C. H. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2001--2010
2017
-
[23]
K.; Kim, J.; and Kim, J
Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017. Continual learning with deep generative replay. Advances in neural information processing systems, 30
2017
-
[24]
Shmelkov, K.; Schmid, C.; and Alahari, K. 2017. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE international conference on computer vision, 3400--3409
2017
-
[25]
Yang, G.; Fini, E.; Xu, D.; Rota, P.; Ding, M.; Nabi, M.; Alameda-Pineda, X.; and Ricci, E. 2022. Uncertainty-aware contrastive distillation for incremental semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2): 2567--2581
2022
-
[26]
Yoon, J.; Yang, E.; Lee, J.; and Hwang, S. J. 2018. Lifelong Learning with Dynamically Expandable Networks. In International Conference on Learning Representations
2018
-
[27]
Yu, L.; Twardowski, B.; Liu, X.; Herranz, L.; Wang, K.; Cheng, Y.; Jui, S.; and Weijer, J. v. d. 2020. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6982--6991
2020
-
[28]
Zenke, F.; Poole, B.; and Ganguli, S. 2017. Continual learning through synaptic intelligence. In International conference on machine learning, 3987--3995. PMLR
2017
-
[29]
Zhang, C.-B.; Xiao, J.-W.; Liu, X.; Chen, Y.-C.; and Cheng, M.-M. 2022 a . Representation compensation networks for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7053--7064
2022
-
[30]
Zhang, Z.; Gao, G.; Fang, Z.; Jiao, J.; and Wei, Y. 2022 b . Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. Advances in neural information processing systems, 35: 24340--24353
2022
-
[31]
H.; and Wei, Y
Zhang, Z.; Gao, G.; Jiao, J.; Liu, C. H.; and Wei, Y. 2023. Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 843--853
2023
-
[32]
Zhao, D.; Yuan, B.; and Shi, Z. 2023. Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 11932--11947
2023
-
[33]
Zhao, H.; Wang, H.; Fu, Y.; Wu, F.; and Li, X. 2021. Memory-efficient class-incremental learning for image classification. IEEE Transactions on Neural Networks and Learning Systems, 33(10): 5966--5977
2021
-
[34]
Zhou, B.; Zhao, H.; Puig, X.; Fidler, S.; Barriuso, A.; and Torralba, A. 2017. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, 633--641
2017
-
[35]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[36]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.