Revisiting Continual Semantic Segmentation with Pre-trained Vision Models

Duzhen Zhang; Feilong Chen; Junhao Zheng; Qiaoyi Su; Qi Tian; Shuncheng Jia; Tielin Zhang; Wei Cong; Xuanle Zhao; Ye Bai

Revisiting Continual Semantic Segmentation with Pre-trained Vision Models

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that pre-trained vision models barely forget under direct fine-tuning, and that the forgetting that does occur is driven by classifier drift away from the pretrained backbone, not by degradation of the backbone's representa

desk verdict A clean, provocative empirical claim that DFT is not a lower bound in continual semantic segmentation — but the 'inherent anti-forgetting' generalization needs harder settings before it can be trusted. read the letter →

arxiv 2508.04267 v1 pith:6TL2K4D3 submitted 2025-08-06 cs.CV

Duzhen Zhang , Yong Ren , Wei Cong , Junhao Zheng , Qiaoyi Su , Shuncheng Jia , Zhong-Zhi Li , Xuanle Zhao

show 4 more authors

Ye Bai Feilong Chen Qi Tian Tielin Zhang

This is my paper

classification cs.CV

keywords continualsemanticsegmentationpre-trainedvisionmodelsdirectfine-tuningcatastrophicforgettingclassifierdriftbackbonefreezingDFT*feature-spaceprobing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper revisits continual semantic segmentation (CSS) and challenges a long-held assumption: that direct fine-tuning (DFT) is a weak baseline doomed by catastrophic forgetting. Through probing experiments on Pascal VOC 2012 and ADE20K across eight CSS settings with ResNet101 and Swin-B backbones, the authors find that pre-trained vision models (PVMs) retain previously learned knowledge remarkably well even under DFT. They trace the residual forgetting to the classifier head drifting away from the PVM, while the backbone's feature representations remain stable. Based on this, they propose DFT*, a stripped-down strategy that freezes the backbone and previously learned classifiers and pre-allocates future classifiers, matching or beating sixteen state-of-the-art CSS methods with far fewer trainable parameters and less training time. If this claim holds, much of the complexity in modern CSS methods may be unnecessary.

What carries the argument

The central object is the pre-trained vision model (PVM) used as a backbone in continual semantic segmentation. The paper's argument rests on a feature-space probing analysis that separates the contributions of the backbone and the classifier to catastrophic forgetting. The key mechanism is the observation that during direct fine-tuning, backbone representations remain stable across tasks while the classifier's decision boundary drifts away from those stable features. DFT* operationalizes this by freezing the backbone and previously learned classifiers and pre-allocating future classifiers, thereby preventing classifier drift while preserving the PVM's knowledge.

What would settle it

Run DFT on a continual segmentation benchmark where new classes are visually similar to old classes — for example, splitting a semantic category like 'vehicle' into 'car', 'truck', 'bus' across tasks — and measure old-class IoU after each step. If forgetting rises sharply, the claim of inherent anti-forgetting fails. Alternatively, directly measure backbone feature drift (e.g., CKA distance between old-class features before and after training on new classes) in such a confusable-class setting; the paper's claim predicts near-zero drift, so a large drift would falsify it.

Watch

Extended reading notes

Core claim

The central discovery is that existing CSS methods significantly underestimate the inherent anti-forgetting capabilities of pre-trained vision models. Even under direct fine-tuning — sequentially fine-tuning the model across class sets with no special mitigation — PVMs retain previously learned knowledge with minimal forgetting. A detailed feature-space probing analysis shows that the observed forgetting is primarily caused by the classifier's drift away from the PVM, rather than by degradation of the backbone representations. This insight leads to DFT*, a simple enhancement of DFT that freezes the PVM backbone and previously learned classifiers while pre-allocating future classifiers. DFT*

Load-bearing premise

The conclusion that PVMs have inherent anti-forgetting capabilities rests on the assumption that the eight CSS settings across Pascal VOC 2012 and ADE20K with ResNet101 and Swin-B are representative; if those settings happen to be unusually forgiving, the low forgetting observed under DFT would not transfer to harder continual segmentation setups.

Editorial extensions

If this is right

If PVMs truly have inherent anti-forgetting capabilities, then many complex continual learning techniques — replay, regularization, distillation — may be unnecessary for CSS and could be replaced by simpler parameter-freezing or classifier-only strategies.
DFT*'s competitive performance with far fewer trainable parameters suggests a favorable accuracy-efficiency trade-off, making continual segmentation more practical in resource-constrained settings.
The finding that forgetting stems from classifier drift rather than backbone degradation points to a targeted fix: keeping the classifier aligned with the pretrained feature space, e.g., by freezing or by pre-allocating heads.
Pre-allocating future classifiers allows the model to accommodate new classes without modifying the frozen backbone, simplifying the continual-learning protocol to a classifier-growing problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to test whether the same 'backbone-stable, classifier-drift' pattern holds for other pretrained vision tasks such as continual object detection or continual instance segmentation, where the heads are typically more complex than a single linear classifier.
The paper's focus on standard benchmarks may understate forgetting when new classes are semantically similar to old ones (e.g., splitting a superclass into subclasses); a stress test with high class overlap could reveal limits of the 'inherent anti-forgetting' claim.
If classifier drift is the sole culprit, then a testable implication is that merely re-learning the classifier on a small buffer of old examples after each task should fully restore old-class performance without touching the backbone — an experiment the paper's logic would predict to succeed.
The pre-allocation of future classifiers implies the number of classes is known in advance; relaxing this to open-set or unbounded class streams would require an adaptive head-growing mechanism, which the paper leaves implicit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a clean, provocative empirical claim — DFT is not the lower bound, and the forgetting in continual semantic segmentation lives in the classifier, not the backbone. The paper is abstract-only as of now, so everything is provisional, but the framing is fresh and the proposed DFT* is a sensible test.

What's actually new: the paper systematically probes where forgetting occurs across eight settings and two backbones (ResNet101, Swin-B), and it challenges the long-standing assumption that DFT is a performance floor. That's a useful reset. The DFT* recipe — freeze the backbone and old classifiers, preallocate future classifiers — is not conceptually exotic, but the diagnostic framing is, and the claim that it beats sixteen SOTA methods with far fewer trainable parameters is worth checking.

The soft spots: the main one is generalizability. The abstract doesn't say whether the eight settings include overlapped and disjoint splits, how hard the incremental orderings are, or how much class overlap exists. In CSS, disjoint splits can be forgiving — old and new classes never appear together, so the backbone isn't pressured to overwrite. If DFT's low forgetting is shown mostly on easy disjoint settings, the 'inherent anti-forgetting' claim is overstated. The stress-test note makes this point and I think it lands. The paper needs to show at least one high-overlap or adversarially ordered setting to support the sweeping conclusion. Also, since only the abstract is available, there's no way to check statistical significance, hyperparameter sensitivity, or whether the preallocated classifiers introduce any bias. The authors should report those details and, ideally, release code.

The citation pattern: can't judge from the abstract alone. No obvious red flag.

Bottom line: this is a paper that deserves a serious referee. The central claim is plausible but not yet established. A referee should push for evidence across more diverse settings and for more transparency about the splits. If the results hold up, this could simplify a lot of the CSS literature.

Referee Report

4 major / 3 minor

Summary. The paper investigates continual semantic segmentation (CSS) and challenges the common assumption that direct fine-tuning (DFT) of pre-trained vision models (PVMs) suffers from severe catastrophic forgetting. Based on experiments on Pascal VOC 2012 and ADE20K under eight CSS settings with ResNet101 and Swin-B, the authors report that PVMs retain previous knowledge with minimal forgetting under DFT, and that the observed forgetting mainly stems from classifier drift rather than backbone degradation. They propose DFT*, which freezes the PVM backbone and previously learned classifiers and pre-allocates future classifiers, and claim it is competitive or superior to sixteen state-of-the-art CSS methods while training fewer parameters.

Significance. If the central claim holds, the paper would reframe the CSS problem: instead of designing complex strategies to protect backbone features, simple freezing of the backbone plus classifier management may suffice. This would be a practically important simplification and could reduce the field's reliance on heavier methods. The proposed baseline may also serve as a stronger, more meaningful lower bound for future CSS research. However, the available evidence is only an abstract; the claim's validity depends on the representativeness of the experimental settings and the statistical rigor of the comparisons, neither of which can be assessed from the abstract alone.

major comments (4)

[Abstract (Pascal VOC/ADE20K, eight CSS settings)] The central claim that PVMs have 'inherent anti-forgetting capabilities' rests on eight CSS settings on two datasets with two backbones. The abstract does not state whether the settings are disjoint or overlapped, the ordering of incremental tasks, the class overlap between tasks, or how class similarity is distributed. In CSS, these choices strongly determine forgetting: disjoint splits with low class overlap can make the problem artificially easy, whereas overlapped or adversarially ordered splits induce direct competition between old and new classes. Without a demonstration that the settings are representative and include difficult cases, the generalization from 'these settings' to 'inherent anti-forgetting' is not established. This is a load-bearing issue because the headline conclusion is exactly about inherentness.
[Abstract (comparison with 16 methods)] The abstract reports that DFT* achieves competitive or superior performance compared to sixteen state-of-the-art CSS methods, but it gives no statistical significance information, confidence intervals, or variance across runs. Given that the difference between a simple baseline and complex methods may be small, the claim of superiority or competitiveness could be within run-to-run noise. The paper should report standard deviations and ideally significance tests for the main comparisons; otherwise the quantitative claim is not verifiable from the abstract and, if omitted in the full paper, would undermine the conclusion.
[Abstract (DFT*: pre-allocating future classifiers)] The proposed DFT* freezes the backbone and previously learned classifiers and pre-allocates future classifiers. A key unstated detail is how the number of pre-allocated classifiers is chosen and whether this choice uses future-task information (e.g., the total number of classes or the class names). In continual learning benchmarks, knowing the total number of classes ahead of time can leak information that is not available in a truly open-ended setting. If the pre-allocation number is tuned on the benchmark or set using future knowledge, the comparison to methods that do not make this assumption would be unfair. The abstract should at least clarify whether this is a free parameter and how it is selected.
[Abstract (probing analysis)] The paper's second major conclusion is that 'the observed forgetting primarily arises from the classifier's drift away from the PVM, rather than from degradation of the backbone representations.' This is derived from a 'detailed probing analysis' that is not described in the abstract. Without knowing the probe methodology (e.g., linear probing on frozen features, SVM, or a diagnostic classifier), it is hard to judge whether the result reflects a property of the PVM or an artifact of the probe. Please include a summary of the probe design and its sensitivity in the full paper.

minor comments (3)

[Abstract] The term 'eight CSS settings' is used without enumerating them; listing them (e.g., disjoint vs. overlapped, 15-5, 10-5, etc.) would make the scope of the claim more transparent.
[Abstract] The phrase 'two representative PVM backbones' is slightly vague; 'representative' could be justified by stating that ResNet101 is a classic CNN and Swin-B is a modern ViT-based backbone.
[Abstract] DFT* is introduced with an asterisk but never unpacked; a brief definition or expansion would help readers unfamiliar with the shorthand.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: abstract-only empirical study with no derivation that reduces to its inputs.

full rationale

This is an abstract-only review. The paper is an empirical study: it reports probing analyses of Direct Fine-Tuning (DFT) on Pascal VOC 2012 and ADE20K under eight settings and proposes DFT* based on the observed stability of PVM backbone features. There is no equation or fitted parameter in the abstract that is later renamed as a prediction, no self-citation is invoked as load-bearing evidence, and no uniqueness theorem or ansatz is smuggled in via citation. The central claim that PVMs have inherent anti-forgetting capabilities is an empirical generalization, not a definitional or statistical artifact. Even though the proposed method is validated on the same benchmarks that motivated the analysis, that is a normal experimental loop, not circular reasoning: the benchmark results are external data, not constructed from the method's own output. Under the hard rule that circularity requires a quotable reduction (Eq. X = Eq. Y by construction, or fitted parameter renamed as prediction), no such reduction is present in the available text. The correct finding is therefore no significant circularity, score 0. Concerns about generalizability across task order, overlap, and class difficulty are correctness/validity risks, not circularity, and cannot be evaluated from the abstract alone.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces a method (DFT*) rather than new physical entities. The main assumptions are about benchmark representativeness, the reliability of the probing analysis, and the generality of the two chosen backbones.

free parameters (2)

number of pre-allocated future classifiers = not specified in abstract
DFT* pre-allocates classifiers for future classes; the count or capacity is likely a hyperparameter, but no value is given in the abstract.
hyperparameters of DFT* (learning rate, epochs, optimizer) = not specified in abstract
Standard training hyperparameters must be set to reproduce the results; not disclosed in the abstract.

assumptions (3)

domain assumption Pascal VOC 2012 and ADE20K, and the eight chosen CSS settings, are representative of continual semantic segmentation.
The paper generalizes from these benchmarks/settings to a claim about PVMs in general ('inherent anti-forgetting capabilities'). If these settings are easy or unrepresentative, the conclusion may not transfer.
domain assumption The probing analysis distinguishes classifier drift from backbone degradation accurately.
The claim that forgetting is 'primarily' from classifier drift depends on the probes separating the two sources. The abstract does not specify the probe metrics.
domain assumption Pre-trained vision models used (ResNet101, Swin-B) are representative of PVMs.
The conclusion about 'PVMs' is drawn from two specific backbones; other architectures may behave differently.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revisiting Continual Semantic Segmentation with Pre-trained Vision Models." pith.science (2026). https://pith.science/paper/6TL2K4D3

@misc{pith2026250804267,
  author       = {Pith},
  title        = {Pith review of: Revisiting Continual Semantic Segmentation with Pre-trained Vision Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TL2K4D3}},
  note         = {Machine review of arXiv:2508.04267}
}

read the original abstract

Continual Semantic Segmentation (CSS) seeks to incrementally learn to segment novel classes while preserving knowledge of previously encountered ones. Recent advancements in CSS have been largely driven by the adoption of Pre-trained Vision Models (PVMs) as backbones. Among existing strategies, Direct Fine-Tuning (DFT), which sequentially fine-tunes the model across classes, remains the most straightforward approach. Prior work often regards DFT as a performance lower bound due to its presumed vulnerability to severe catastrophic forgetting, leading to the development of numerous complex mitigation techniques. However, we contend that this prevailing assumption is flawed. In this paper, we systematically revisit forgetting in DFT across two standard benchmarks, Pascal VOC 2012 and ADE20K, under eight CSS settings using two representative PVM backbones: ResNet101 and Swin-B. Through a detailed probing analysis, our findings reveal that existing methods significantly underestimate the inherent anti-forgetting capabilities of PVMs. Even under DFT, PVMs retain previously learned knowledge with minimal forgetting. Further investigation of the feature space indicates that the observed forgetting primarily arises from the classifier's drift away from the PVM, rather than from degradation of the backbone representations. Based on this insight, we propose DFT*, a simple yet effective enhancement to DFT that incorporates strategies such as freezing the PVM backbone and previously learned classifiers, as well as pre-allocating future classifiers. Extensive experiments show that DFT* consistently achieves competitive or superior performance compared to sixteen state-of-the-art CSS methods, while requiring substantially fewer trainable parameters and less training time.

Discussion (0). Sign in to comment.

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.