REVIEW 4 major objections 4 minor 31 references
Stabilizing Open-Set Test-Time Adaptation via Primary-Auxiliary Filtering and Knowledge-Integrated Prediction
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that open-set test-time adaptation is stabilized by a primary-auxiliary filtering scheme and a confidence-weighted three-model prediction ensemble, yielding gains in both closed-set accuracy and open-set discrimination acro
desk verdict A solid, incremental OSTTA method with broad experiments, but the abstract overclaims 'both' metrics — one table shows a small ACC deficit — and there are no error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Primary-Auxiliary Filtering (PAF): a two-filter gate. The primary filter is the adapting model's entropy threshold (below tau = 0.4*log(C) counts as closed-set), which captures current target-domain knowledge; the auxiliary filter is the EMA model's entropy, which provides stability. The two filters combine hard gating for entropy maximization (applied only when both reject) with a soft weight w_soft(x)=1/exp(H(f_EMA(~x))-tau) for entropy minimization. Knowledge-Integrated Prediction (KIP): a per-sample weighted average of the logits of source, adapting, and EMA models, with weight c_i(x)=1/3+gamma*(max(p_i(x)) - average max probability), up-weighting whichever model is most confident for th
What would settle it
Measure KIP against simple logit averaging and each single model on a domain shift the source model has never seen, such as a corruption type absent from its training corruptions. If the ensemble's H-score falls below the best single model on closed-set samples where the source model is confidently wrong, the claimed complementarity fails. Alternatively, artificially increase source-model miscalibration (for example by temperature-scaling it away from 1) and check whether KIP's gain shrinks or reverses.
Extended reading notes
Core claim
The central claim is that OSTTA can be stabilised without replay buffers or heavy augmentation by separating the filtering problem from the inference problem. For filtering, the paper shows that using only the adapting model's entropy to decide which samples get entropy minimization versus entropy maximization accumulates errors, while using only the EMA model lags behind domain shifts. PAF therefore runs both: the adapting model proposes a label, and the EMA model vetoes entropy maximization or down-weights entropy minimization. Only samples both filters reject receive the entropy-maximising loss; samples the primary filter accepts receive entropy-minimising loss weighted by the EMA model's
Load-bearing premise
The load-bearing premise is that the frozen source model, even when miscalibrated on the shifted target domain, contributes complementary knowledge to the final ensemble; if the source model becomes confidently wrong on target-domain closed-set samples, KIP's confidence weighting could amplify that error instead of correcting it.
Editorial extensions
If this is right
- The method is buffer-free: it needs only the current batch and EMA parameters, so it can run in privacy-sensitive streaming settings without replay memory.
- Filtering and inference should be treated as separate problems: the EMA model is most useful as a stabiliser of the primary filter, not as the sole decision maker.
- A frozen source model can still rescue closed-set predictions on samples the adapting model has been trained to distrust, because it has never been exposed to open-set samples.
- Only batch-normalization layers need updating, keeping inference cheap: 0.083 s per batch on CIFAR100-C with SVHN-C, compared with 0.626 s for STAMP.
- The benefit of confidence-weighted ensembling over naive averaging grows with prolonged open-set exposure: the appendix reports up to a 1 percentage-point accuracy-gap advantage under 60 domain shifts.
Reading between the lines
- Editorial inference: the same primary-auxiliary gate could be applied to other self-training signals, such as pseudo-labels or contrastive objectives, since the instability it fixes is generic to student-teacher loops.
- Editorial inference: the results suggest the source model acts as a regulariser that resists the adapting model's open-set drift; a natural stress test is to measure KIP's gain when the source model is deliberately miscalibrated on the target domain, a case the paper does not report.
- Editorial inference: PAF's exclusion rule (primary says open-set, auxiliary says closed-set) is a conservative middle path whose effect on open-set recall is not isolated per domain; analysing which samples get excluded would clarify when this rule helps versus when it simply withholds training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies Open-set Test-Time Adaptation (OSTTA), where the test stream contains both closed-set samples under domain shift and unknown open-set samples. It proposes two components: Primary-Auxiliary Filtering (PAF), which uses the adapting model as a primary filter and an EMA model as an auxiliary filter to decide which samples receive entropy minimization versus entropy maximization; and Knowledge-Integrated Prediction (KIP), which ensembles the logits of the source, adapting, and EMA models with per-sample confidence-based weights. The method is evaluated on CIFAR10-C, CIFAR100-C, and ImageNet-C as closed-set data with SVHN-C, TinyImageNet-C, Places365-C, and Textures-C as open-set data, compared against nine TTA/OSTTA baselines, and supplemented with ablations, hyperparameter sensitivity, latency, batch-size, and open-set-ratio experiments.
Significance. The problem is practically important and the empirical package is unusually broad: three closed-set and four open-set benchmarks, nine baselines, ablations of each proposed component, hyperparameter sensitivity, smaller-batch robustness, open-set-ratio robustness, and inference latency. The method is lightweight, updates only batch-normalization layers, and does not require a replay buffer, which is attractive for privacy-sensitive streaming settings. The code is released. If the reported results are reproducible, PAF-KIP would be a strong and practical baseline for OSTTA. The main weakness is that the headline claim is stated more strongly than the authors' own tables support, and no uncertainty quantification is provided for any of the reported point estimates.
major comments (4)
- [Abstract and Section 3.2, Table 2] The abstract states that the method 'enhances both closed-set accuracy and open-set discrimination over existing methods.' This is directly contradicted by Table 2 for CIFAR100-C with TinyImageNet-C as the open-set dataset: the proposed method obtains ACC 63.79, which is 0.46 points below STAMP's 64.25. The method wins on H-score only because its AUR is much higher (84.16 vs. 73.16). Either the claim should be restricted to H-score / 'in most settings', or the authors should provide statistical evidence that the 0.46 deficit is within noise. As written, the headline claim is false for a configuration reported in the paper's own main table.
- [Section 3.1-3.2, Tables 1-3 and Appendix Tables 7-8] No error bars, standard deviations, confidence intervals, or statistical significance tests are reported anywhere. All numbers are point estimates averaged over 15 domains, and several claimed improvements are very small (e.g., Table 2, Places365 row: ACC 62.94 vs. 62.89 for STAMP, a +0.05 difference). Without multiple seeds or per-domain paired comparisons, the claim of consistent superiority over prior methods is not supported. This is load-bearing because the central contribution is empirical. At minimum, report mean±std over at least three seeds and, where possible, paired tests across the 15 corruption domains.
- [Section 2.4, Eqs. (10)-(11), and Figure 2] KIP assumes that the source model, although not adapted to the target domain, contributes complementary knowledge because it was never exposed to open-set data. The weights in Eq. (10) increase a model's contribution when its maximum class probability is above the ensemble average. If the source model is confidently wrong or severely miscalibrated on a target domain, this weighting could amplify harmful logits rather than correct them. The paper does not analyze failure cases or conditions under which KIP might degrade relative to simple logit averaging or to using only the adapting model. A discussion or a targeted experiment (e.g., stronger corruptions, source model with lower clean accuracy) would strengthen the claim that KIP is broadly beneficial rather than only helpful in the tested settings.
- [Appendix Table 8] The open-set-ratio experiments contain a striking anomaly: for STAMP on CIFAR100-C, AUR is 47.58 at ratio 0.50 but jumps to 90.21 at ratio 0.75, then remains near 91 for higher ratios. This non-monotonic discontinuity is not discussed. Either it reflects genuine instability of the baseline under this setting, which should be explained, or it indicates a reporting/implementation issue. Since this table is used to claim robustness across open-set ratios, the anomaly should be addressed rather than left unexplained.
minor comments (4)
- [Section 3.2, Table 2 discussion] The text says the method 'achieves performance improvements across most cases', which is more cautious than the abstract's 'enhances both'. The abstract should be harmonized with the more precise language used in the results section.
- [Appendix Algorithm 1] The pseudo-code formatting makes lines 6-11 confusing: after 'if Fpr(xi)=1 then' there is a separate 'if Fpr(xi)=0 then' block at the same indentation. It is logically correct as independent if statements, but indentation and an explicit 'else' would improve readability.
- [Section 2.4, Eq. (10)] The term 'calibrates' is potentially misleading: Eq. (10) re-weights logits by maximum class probability but does not perform probability calibration in the standard sense. Suggest using 're-weights' or 'normalizes' instead.
- [Appendix Table 8] The table rows are typeset with values run together and no spacing after commas (e.g., '88.91+1.8596.45+4.8692.53+4.51'). This makes the table hard to read and should be reformatted.
Circularity Check
No circularity: PAF and KIP are defined by independent equations, hyperparameters are taken from prior work or heuristics, and the reported benchmark results are measured rather than constructed from fitted parameters.
full rationale
The paper's derivation chain is self-contained. Primary-Auxiliary Filtering (Eqs. 6-9) uses the adapting and EMA models' entropies with a fixed threshold tau = 0.4*log(C) following external prior work [21]; no parameter is fitted to the evaluation metrics. Knowledge-Integrated Prediction (Eqs. 10-11) is a fixed confidence-weighted ensemble of source, adapting, and EMA logits, with gamma set to 0.1 and no fitting to test labels. The central claim is supported by direct benchmark measurements (Tables 1-3) and ablations (Table 4, Table 5, Figure 2, Figure 6). There are no self-citations, no imported uniqueness theorems, no renamed known results, and no fitted quantity that is later called a prediction. The abstract's unqualified 'enhances both' statement is contradicted in one configuration by the paper's own Table 2 (CIFAR100-C + TinyImageNet-C: ACC 63.79 vs STAMP 64.25), but that is an empirical overstatement or correctness risk, not circularity: the result remains measured, not derived from its own inputs. Similarly, the possibility that the source model is confidently wrong in KIP is a limitation, not a circular step, because KIP's weights are a fixed heuristic whose behavior is ablated rather than fitted to force a desired outcome.
Assumptions & free parameters
free parameters (4)
- loss balancing hyperparameter α =
2.0 (CIFAR), 0.7 (ImageNet)
- EMA decay rate β =
0.999
- KIP sharpness γ =
0.1
- confidence threshold τ =
0.4 × log(C)
assumptions (5)
- domain assumption The test stream contains a mixture of closed-set and open-set samples in a 1:1 ratio in the main experiments.
- domain assumption Entropy minimization on closed-set samples and entropy maximization on open-set samples improves OSTTA.
- domain assumption The source model, never adapted to the target domain, retains complementary knowledge that improves the final ensemble.
- domain assumption The EMA model provides a stable reference that prevents the rapid error accumulation of the adapting model.
- domain assumption Updating only batch normalization layers is sufficient for test-time adaptation.
Cite this review
Pith. "Pith review of Stabilizing Open-Set Test-Time Adaptation via Primary-Auxiliary Filtering and Knowledge-Integrated Prediction." pith.science (2026). https://pith.science/paper/WSCMIWEV
@misc{pith2026250818751,
author = {Pith},
title = {Pith review of: Stabilizing Open-Set Test-Time Adaptation via Primary-Auxiliary Filtering and Knowledge-Integrated Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/WSCMIWEV}},
note = {Machine review of arXiv:2508.18751}
}
read the original abstract
Deep neural networks demonstrate strong performance under aligned training-test distributions. However, real-world test data often exhibit domain shifts. Test-Time Adaptation (TTA) addresses this challenge by adapting the model to test data during inference. While most TTA studies assume that the training and test data share the same class set (closed-set TTA), real-world scenarios often involve open-set data (open-set TTA), which can degrade closed-set accuracy. A recent study showed that identifying open-set data during adaptation and maximizing its entropy is an effective solution. However, the previous method relies on the source model for filtering, resulting in suboptimal filtering accuracy on domain-shifted test data. In contrast, we found that the adapting model, which learns domain knowledge from noisy test streams, tends to be unstable and leads to error accumulation when used for filtering. To address this problem, we propose Primary-Auxiliary Filtering (PAF), which employs an auxiliary filter to validate data filtered by the primary filter. Furthermore, we propose Knowledge-Integrated Prediction (KIP), which calibrates the outputs of the adapting model, EMA model, and source model to integrate their complementary knowledge for OSTTA. We validate our approach across diverse closed-set and open-set datasets. Our method enhances both closed-set accuracy and open-set discrimination over existing methods. The code is available at https://github.com/powerpowe/PAF-KIP-OSTTA .
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
A probabilistic framework for lifelong test-time adaptation
Dhanajit Brahma and Piyush Rai. A probabilistic framework for lifelong test-time adaptation. In CVPR, pages 3582–3591, 2023
work page 2023
-
[2]
Contrastive test-time adaptation
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, pages 295–305, 2022
work page 2022
-
[3]
Describing textures in the wild
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606–3613, 2014
work page 2014
-
[4]
Robustbench: a standardized adversarial robustness benchmark
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In NIPS workshop on Datasets and Benchmarks Track, 2021
work page 2021
-
[5]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009
2009
-
[6]
Robust mean teacher for continual and gradual test-time adaptation
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In CVPR, pages 7704–7714, 2023. LEE ET AL.: STABILIZING OPEN-SET TTA VIA PAF AND KIP 11
work page 2023
-
[7]
Sharpness- aware minimization for efficiently improving generalization
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020
arXiv 2010
-
[8]
Decorate the newcomers: Visual domain prompt for continual test time adaptation
Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI, volume 37, pages 7595–7603, 2023
work page 2023
Show all 31 references
-
[9]
Unified entropy optimization for open-set test-time adaptation
Zhengqing Gao, Xu-Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. In CVPR, pages 23975–23984, 2024
2024
-
[10]
Recent advances in open set recognition: A survey
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. PAMI, 43(10):3614–3631, 2020
2020
-
[11]
Note: Robust continual test-time adaptation against temporal correlation
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. NIPS, 35:27253–27266, 2022
2022
-
[12]
Sotta: Robust test-time adaptation on noisy data streams
Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. NIPS, 36, 2024
2024
-
[13]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016
2016
-
[14]
Membn: Robust test-time adaptation via batch norm with statistics memory
Juwon Kang, Nayeong Kim, Jungseul Ok, and Suha Kwak. Membn: Robust test-time adaptation via batch norm with statistics memory. In ECCV, pages 467–483. Springer, 2025
2025
-
[15]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[16]
Tiny imagenet visual recognition challenge
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS231N, 7(7):3, 2015
2015
-
[17]
Towards open-set test- time adaptation utilizing the wisdom of crowds in entropy minimization
Jungsoo Lee, Debasmit Das, Jaegul Choo, and Sungha Choi. Towards open-set test- time adaptation utilizing the wisdom of crowds in entropy minimization. In ICCV, 2023
2023
-
[18]
A comprehensive survey on test-time adaptation under distribution shifts
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. IJCV, pages 1–34, 2024
2024
-
[19]
Vida: Homeostatic visual domain adapter for continual test time adaptation
Jiaming Liu, Senqiao Yang, Peidong Jia, Renrui Zhang, Ming Lu, Yandong Guo, Wei Xue, and Shanghang Zhang. Vida: Homeostatic visual domain adapter for continual test time adaptation. arXiv preprint arXiv:2306.04344, 2023
2023 arXiv
-
[20]
Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2011, page 4. Granada, 2011
2011
-
[21]
Efficient test-time model adaptation without forgetting
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, 2022. 12 LEE ET AL.: STABILIZING OPEN-SET TTA VIA PAF AND KIP
2022
-
[22]
Test: Test-time self-training under distribution shift
Samarth Sinha, Peter Gehler, Francesco Locatello, and Bernt Schiele. Test: Test-time self-training under distribution shift. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pages 2759–2769, 2023
2023
-
[23]
Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. NIPS, 30, 2017
2017
-
[24]
Un- mixing test-time normalization statistics: Combatting label temporal correlation
Devavrat Tomar, Guillaume Vray, Jean-Philippe Thiran, and Behzad Bozorgtabar. Un- mixing test-time normalization statistics: Combatting label temporal correlation. arXiv preprint arXiv:2401.08328, 2024
2024 arXiv
-
[25]
Tent: Fully test-time adaptation by entropy minimization
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021
2021
-
[26]
Continual test-time domain adaptation
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, 2022
2022
-
[27]
Stamp: Outlier-aware test-time adaptation with stable memory replay
Yongcan Yu, Lijun Sheng, Ran He, and Jian Liang. Stamp: Outlier-aware test-time adaptation with stable memory replay. In ECCV, pages 375–392. Springer, 2025
2025
-
[28]
Robust test-time adaptation in dynamic scenarios
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, 2023
2023
-
[29]
Wide residual networks
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016
2016
-
[30]
Delta: Degradation-free fully test-time adaptation
Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: Degradation-free fully test-time adaptation. In ICLR
-
[31]
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. PAMI, 40(6):1452–1464, 2017. LEE ET AL.: STABILIZING OPEN-SET TTA VIA PAF AND KIP 1 A Related Works Test-Time Adaptation Test-time adaptatio...
2017
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.