REVIEW 3 major objections 6 minor 33 references
Segmentation Assisted Incremental Test Time Adaptation in an Open World
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A background-pixel count reveals unseen classes during adaptation
desk verdict Useful new ITTA protocol and a simple training-free plug-in that mostly works; the one real hole is the never-ablated alpha=0.95 threshold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
SegAssist is the central object: a training-free plug-in that turns CLIP's patch features into a segmentation map over the set topK(xt) ∪ {background}, then computes the proportion of pixels labeled background. The threshold α = 0.95 on this background ratio is the decision rule that decides whether an uncertain sample is worth an oracle query. The companion metric ICDD (Incremental Class Detection Delay) is the area between the cumulative ground-truth curve of introduced classes and the cumulative detected-class curve, quantifying timeliness.
What would settle it
A concrete test: take a new domain (e.g., a weather-shifted driving scene) and run the ITTA protocol with and without SegAssist, measuring the precision of selected samples at identifying truly unseen classes. If at α=0.95 the selected-sample precision is not higher than MSP's, or if lowering α to 0.8 changes harmonic-mean accuracy substantially, the background-ratio premise fails.
Extended reading notes
Core claim
The central claim is that CLIP's local, patch-level features already contain enough signal to tell an uncertain seen-class image from an uncertain unseen-class image, even though global softmax scores overlap heavily. SegAssist computes a per-pixel segmentation of an uncertain image using the top-5 predicted class names plus the literal class 'background', then measures the background ratio B(xt). If B(xt) > 0.95, the sample is sent to the oracle. The paper reports that this filter raises harmonic-mean accuracy across ZS-Eval, TDA, and DPE baselines and lowers the newly defined Incremental Class Detection Delay on ImageNet-R, ImageNet-A, and DomainNet clipart/painting, compared with random,
Load-bearing premise
The claim that a background ratio above 0.95 marks unseen-class samples is the load-bearing step; it is supported only by a motivating statistic using a looser 'majority background' definition, and the 0.95 threshold itself is never varied or analyzed.
Editorial extensions
If this is right
- Existing single-image CLIP-based TTA methods (ZS-Eval, TDA, DPE) can be upgraded to open-world ITTA by adding SegAssist without retraining, since it uses only features already computed by CLIP.
- The 1% replenishing budget protocol provides a common benchmark for comparing active-labeling strategies under online streaming, where batch-level statistics are unavailable.
- ICDD gives a second axis for evaluating ITTA systems, so methods that detect classes late but classify accurately are no longer conflated with methods that detect them early.
- Since SegAssist improves both harmonic-mean accuracy and ICDD in most reported settings, the background-ratio heuristic appears to generalize across different TTA backbones and four domain-shifted datasets.
- The module's independence from the TTA method means gains from SegAssist and gains from better TTA should add, allowing modular progress on active selection and adaptation separately.
Reading between the lines
- Inference: The paper's motivating statistic uses 'majority background pixels' while deployment uses >95%, so a natural next experiment is to ablate α to see whether the filter is selecting on background dominance or on an even more extreme tail, and whether the optimum α transfers across datasets.
- Inference: Because SegAssist relies on CLIP's pretrained semantic alignment, its usefulness should degrade for unseen classes that are semantically close to seen classes (e.g., a new bird species among known birds), where patch predictions may land on related seen classes instead of background; the reported cases where SegAssist trails MSP are consistent with this.
- Inference: The ICDD metric could be reused beyond active labeling, e.g., to evaluate continual-learning systems in streaming settings where the same ground-truth introduction schedule is available.
- Inference: A testable extension is to replace the fixed background threshold with a per-image adaptive threshold based on the distribution of patch-vs-text similarities, which might make SegAssist robust to new domains without re-tuning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a new setting called Incremental Test-Time Adaptation (ITTA) for vision-language models, in which the test stream contains both covariate shifts and novel classes. The authors propose a benchmark that combines single-image TTA methods (ZS-Eval, TDA, DPE) with active labeling, and a new plug-in module, SegAssist, that filters uncertain samples by computing a CLIP patch-level segmentation map and selecting only images whose pixels are predominantly classified as 'background' (B(x_t) > 0.95). They also define a metric, Incremental Class Detection Delay (ICDD), that measures how promptly new classes are discovered. Experiments on ImageNet-R, ImageNet-A, and DomainNet Clipart/Painting report harmonic-mean accuracy and ICDD; SegAssist is claimed to improve over random, entropy, and MSP active selection in most configurations.
Significance. The paper proposes a genuinely useful protocol: combining TTA and active labeling for incremental open-world adaptation is a natural and timely benchmark, and the proposed ICDD metric is a reasonable way to separate detection timeliness from classification accuracy. The SegAssist idea of reusing CLIP patch features as a training-free filter is attractive, and the plug-in design means it can be tested with several TTA methods. The empirical gains, however, are small and not accompanied by error bars or multiple seeds, and the key threshold of the method is not validated. If the authors can supply the missing ablations and statistical support, the framework could be a solid contribution.
major comments (3)
- [§5.2, Eq. (2); Fig. 2b; Table 4b] The entire SegAssist selection rule is B(x_t) > α with α = 0.95, but the paper never ablates α. The motivating evidence in Fig. 2b is a 'majority background' statistic (B > 0.5), not B > 0.95. Table 4b varies only the MSP uncertainty threshold, not α; Table 6 varies topK. The exact equality in Table 4b at threshold 0.1 on ImageNet-R (52.14/29.39 with and without SegAssist) suggests that the operating point may be selecting no samples at all in that configuration, but no selected-sample counts or precision/recall numbers are reported. This is load-bearing because SegAssist reduces to this filter. I request an α sweep, the number of queried samples at each α, the fraction of queried samples that truly belong to unseen classes, and ideally validation on a held-out split/domain not used to derive the statistic.
- [Tables 2–4; §7.4] The central claim that SegAssist 'consistently outperforms' MSP is not statistically supported. The gains are mostly small (0.1–3 HM points) and some configurations go the other way: Table 2 ZS-Eval ImageNet-A HM is 23.94 for SegAssist vs 24.59 for MSP; Table 3 TDA Painting ICDD is 41.27 vs 39.31, i.e., SegAssist is worse. No error bars, confidence intervals, or multiple seeds are reported. Because the test stream is fixed and the method is deterministic except for random selection, paired comparisons over several seeds would be feasible. Without such analysis, the conclusion of a consistent improvement should be softened or restricted to the specific configurations where the gap is non-negligible.
- [§4.1, §5.2, Fig. 3] The paper says that when the oracle confirms an unseen class u, the model updates its classifier by adding the text embedding of u. This assumes that a textual class label/prompt is available at query time. In the benchmark datasets this is true by construction because the unseen classes have predefined names. In a genuinely open-world deployment, however, the oracle may provide only a label or a semantic category that is not in a predefined vocabulary. The paper should state this assumption explicitly and address whether the method would require an additional mechanism for obtaining text prompts for discovered classes. Without that clarification, the 'open world' framing is narrower than it appears.
minor comments (6)
- [§5.1 and §5.2] The paper says SegAssist adds 'no computation overhead' in §5.1 and then 'practically no computation overhead' in §5.2. In fact, it performs patch-level scoring against topK+1 text embeddings for every uncertain sample. This is lightweight but not free; reporting wall-clock overhead would be more accurate.
- [Fig. 2b] The y-axis labels (80, 85, 90, 95, 100) are confusing for a 'percentage of background images' plot, since the caption says background images are majority-background. Please clarify the axis and also state that the statistic uses B > 0.5, not the deployed 0.95 threshold.
- [§7.4] The phrase 'only at times second to MSP' in the first paragraph is awkward and could be misread. Please rewrite to say 'only occasionally second to MSP.'
- [Table 4b] The heading 'Varying threshold' should specify that only the MSP/uncertainty threshold is varied; α is held fixed. Otherwise a reader may believe the SegAssist background threshold was ablated.
- [Section 6] The definition of ICDD is given twice in the same section, once in the bullet list and once in the following paragraph. Please consolidate to avoid redundancy.
- [References] Reference [5] contains malformed author text ('Haodong, Dong Lu, Shuo Gong, Jason Wang, and Kristen Moore Xue, Lina Yao'). Please fix the citation; this is likely a formatting error introduced by the bibliography style.
Circularity Check
No significant circularity: SegAssist is an empirically motivated selection heuristic, not a derivation that reduces to its own inputs.
full rationale
The paper's central claim is an experimental comparison: SegAssist, a plug-in that filters uncertain samples by background ratio B(xt)>0.95, improves harmonic-mean accuracy and ICDD when combined with existing TTA methods. This is not a derivation in which the output is identical to the input by construction. The selection rule is explicitly a hypothesis ('We hypothesize that a high proportion of background pixels indicate that xt belongs to an unknown class'), supported by the motivating statistic in Fig. 2b that unseen-class uncertain samples are more often 'background images' (majority background pixels) than seen-class samples. The gap between that statistic (B>0.5) and the deployed threshold (B>0.95) is an empirical design choice, not a fitted parameter that is then renamed as a prediction; no ablation of alpha is reported, but this is a robustness/evaluation concern, not circularity. The ICDD metric is defined externally from ground-truth class-introduction times and is not used in defining B(xt), so the metric is not baked into the selection rule. The paper's self-citations ([1,2,23,24]) are for related TTA and open-set works and are not load-bearing for the SegAssist mechanism; no uniqueness theorem or ansatz is imported from them. The paper even acknowledges failure cases (semantically related unseen classes with B(xt)<alpha), which shows the rule is not tautological. Thus no circular step can be exhibited, and the score is 0.
Assumptions & free parameters
free parameters (4)
- alpha (background ratio threshold) =
0.95
- tau (MSP uncertainty threshold) =
0.2 (default); swept 0.1 to 0.3 in Table 4b
- topK (foreground classes for segmentation) =
5 (default); swept 3, 5, 7 in Appendix A.2
- budget rate r =
1% (10 queries per 1000 samples)
assumptions (4)
- domain assumption The oracle provides correct labels for queried test samples.
- domain assumption CLIP patch features (value vectors from the last attention layer with the final projection layer) support meaningful dense predictions aligned with text embeddings.
- domain assumption The 4:1 seen/unseen split and the chronological order of the test stream mimic real-world open-world label shift.
- domain assumption Existing TTA methods TDA and DPE are used with the hyperparameters from their original papers.
Cite this review
Pith. "Pith review of Segmentation Assisted Incremental Test Time Adaptation in an Open World." pith.science (2026). https://pith.science/paper/XR4Y7LPJ
@misc{pith2026250820029,
author = {Pith},
title = {Pith review of: Segmentation Assisted Incremental Test Time Adaptation in an Open World},
year = {2026},
howpublished = {\url{https://pith.science/paper/XR4Y7LPJ}},
note = {Machine review of arXiv:2508.20029}
}
read the original abstract
In dynamic environments, unfamiliar objects and distribution shifts are often encountered, which challenge the generalization abilities of the deployed trained models. This work addresses Incremental Test Time Adaptation of Vision Language Models, tackling scenarios where unseen classes and unseen domains continuously appear during testing. Unlike traditional Test Time Adaptation approaches, where the test stream comes only from a predefined set of classes, our framework allows models to adapt simultaneously to both covariate and label shifts, actively incorporating new classes as they emerge. Towards this goal, we establish a new benchmark for ITTA, integrating single image TTA methods for VLMs with active labeling techniques that query an oracle for samples potentially representing unseen classes during test time. We propose a segmentation assisted active labeling module, termed SegAssist, which is training free and repurposes the segmentation capabilities of VLMs to refine active sample selection, prioritizing samples likely to belong to unseen classes. Extensive experiments on several benchmark datasets demonstrate the potential of SegAssist to enhance the performance of VLMs in real world scenarios, where continuous adaptation to emerging data is essential. Project-page:https://manogna-s.github.io/segassist/
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
A simple signal for domain shift
Goirik Chakrabarty, Manogna Sreenivas, and Soma Biswas. A simple signal for domain shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023. 1
work page 2023
-
[2]
Santa: Source anchoring network and target alignment for continual test time adaptation
Goirik Chakrabarty, Manogna Sreenivas, and Soma Biswas. Santa: Source anchoring network and target alignment for continual test time adaptation. Transactions on Machine Learning Research, 2023. 1
work page 2023
-
[3]
To- wards unknown-aware learning with virtual outlier synthe- sis
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. To- wards unknown-aware learning with virtual outlier synthe- sis. In International Conference on Learning Representa- tions, page 5, 2022. 3
work page 2022
-
[4]
Active Test-Time Adaptation: Theoretical Analyses and An Algorithm
Shurui Gui, Xiner Li, and Shuiwang Ji. Active test-time adaptation: Theoretical analyses and an algorithm. arXiv preprint arXiv:2404.05094, 2024. 3, 7
work page Pith review arXiv 2024
-
[5]
Learning with mixture of pro- totypes for out-of-distribution detection
Haodong, Dong Lu, Shuo Gong, Jason Wang, and Kris- ten Moore Xue, Lina Yao. Learning with mixture of pro- totypes for out-of-distribution detection. In The Twelfth In- ternational Conference on Learning Representations , 2024. 3
work page 2024
-
[6]
A baseline for detect- ing misclassified and out-of-distribution examples in neural networks
Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016. 3, 7
arXiv 2016
-
[7]
The many faces of robust- ness: A critical analysis of out1-of-distribution generaliza- tion
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out1-of-distribution generaliza- tion. In CVPR, 2021. 8
work page 2021
-
[8]
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021. 8
work page 2021
Show all 33 references
-
[9]
Learning a unified classifier incrementally via rebalancing
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831–839,
-
[10]
Mos: Towards scaling out-of- distribution detection for large semantic space
Rui Huang and Yixuan Li. Mos: Towards scaling out-of- distribution detection for large semantic space. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710–8719, 2021. 3
2021
-
[11]
Efficient test-time adaptation of vision-language models
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14162–14171, 2024. 1, 2, 3, 4, 5, 6, 7, 8
2024
-
[12]
Genetic k-means algo- rithm
K Krishna and M Narasimha Murty. Genetic k-means algo- rithm. IEEE Transactions on Systems, Man, and Cybernet- ics, Part B (Cybernetics), 29(3):433–439, 1999. 3, 7
1999
-
[13]
Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization
Jungsoo Lee, Debasmit Das, Jaegul Choo, and Sungha Choi. Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,
-
[14]
On the robust- ness of open-world test-time training: Self-training with dy- namic prototype expansion
Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robust- ness of open-world test-time training: Self-training with dy- namic prototype expansion. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 2, 3
2023
-
[15]
Learning without forgetting
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelli- gence, 40(12):2935–2947, 2017. 2, 3
2017
-
[16]
Univer- sal test-time adaptation through weight ensembling, diver- sity weighting, and prior correction
Robert A Marsden, Mario D ¨obler, and Bin Yang. Univer- sal test-time adaptation through weight ensembling, diver- sity weighting, and prior correction. In Winter Conference on Applications for Computer Vision, 2024. 2, 3
2024
-
[17]
Moment matching for multi-source domain adaptation
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 1406–1415,
-
[18]
Active domain adaptation via clustering uncertainty-weighted embeddings
Viraj Prabhu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman. Active domain adaptation via clustering uncertainty-weighted embeddings. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8505–8514, 2021. 3, 7
2021
-
[19]
Learn- ing transferable visual models from natural language super- vision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...
2021
-
[20]
icarl: Incremental classifier and representation learning
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017. 2, 3
2001
-
[21]
Improving robustness against common corruptions by covariate shift adaptation
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bring- mann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Neural Information Processing Systems, 2020. 3
2020
-
[22]
Test- time prompt tuning for zero-shot generalization in vision- language models
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. InNeural Information Processing Systems,
-
[23]
Efficient open set single image test time adaptation of vision language models
Manogna Sreenivas and Soma Biswas. Efficient open set single image test time adaptation of vision language models. Transactions on Machine Learning Research, 2025. 2
2025
-
[24]
pstarc: Pseudo source guided target clustering for fully test- time adaptation
Manogna Sreenivas, Goirik Chakrabarty, and Soma Biswas. pstarc: Pseudo source guided target clustering for fully test- time adaptation. In WACV, 2024. 1
2024
-
[25]
Non-parametric outlier synthesis
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. arXiv preprint arXiv:2303.02966, 2023. 3
2023 arXiv
-
[26]
A new active labeling method for deep learning
Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neu- ral networks (IJCNN), pages 112–119. IEEE, 2014. 3
2014
-
[27]
Tent: Fully test-time adaptation by entropy minimization
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. 1, 2, 3
2021
-
[28]
Clipn for zero-shot ood detection: Teaching clip to say no
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 3
2023
-
[29]
Con- tinual test-time domain adaptation
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Con- tinual test-time domain adaptation. In CVPR, 2022. 2, 3
2022
-
[30]
Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation
Monika Wysocza ´nska, Oriane Sim´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation. ECCV, 2024. 5
2024
-
[31]
Active learning for domain adaptation: An energy-based approach
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xin- jing Cheng, and Guoren Wang. Active learning for domain adaptation: An energy-based approach. In Proceedings of the AAAI conference on artificial intelligence , pages 8708– 8716, 2022. 3, 7
2022
-
[32]
Dual prototype evolving for test-time generalization of vision-language models
Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. arXiv preprint arXiv:2410.12790,
-
[33]
Extract free dense labels from clip
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Com- puter Vision (ECCV), 2022. 4, 5 A. Appendix A.1. SegAssist as a plugin module. SegAssist can not only be used with MSP, but also with other standard AL techniques to fu...
2022
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.