Pith. sign in

REVIEW 2 major objections 2 minor 64 references

FRAP improves performance estimation under distribution shift by fusing aligned predictions from a foundation model and the base model to create a reliable reference distribution.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 01:58 UTC pith:2CDA33RA

load-bearing objection FRAP fuses a foundation model into performance estimation via temperature scaling and confidence weighting, but the gains hinge on unexamined complementarity of errors under shift. the 2 major comments →

arxiv 2606.06335 v1 pith:2CDA33RA submitted 2026-06-04 cs.LG cs.AI

Bridging Domain Expertise and Generalization for Performance Estimation

Shuxuan Li , Zhilin Zhao , Quyu Kong , Wei-Shi Zheng This is my paper
classification cs.LG cs.AI
keywords performance estimationdistribution shiftfoundation modelsmodel fusionunlabeled evaluationtemperature scalingconfidence weightingdomain expertise
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Performance estimation under distribution shift is difficult because a model's own outputs grow biased when test data differs from training data, weakening their link to actual accuracy. Prior methods rely only on those outputs and therefore suffer from amplified errors. FRAP instead brings in an external foundation model, aligns its predictions to the base model through temperature-scaled calibration that reduces divergence, and fuses the results with confidence weighting. The resulting reference distribution combines the foundation model's robustness with the base model's domain knowledge, so that the agreement between base-model outputs and this reference serves as the performance estimate. Experiments across datasets and architectures show that this produces consistent gains over existing estimation techniques when labels are unavailable on the shifted test set.

Core claim

FRAP aligns the prediction distribution of the foundation model with that of the base model by applying temperature-scaled calibration that minimizes their divergence. The aligned predictions are fused through confidence-based weighting into a refined reference distribution that integrates robustness from the foundation model and domain-specific expertise from the base model. Performance estimation is then obtained by measuring how closely the base model predictions agree with this reference, yielding a surrogate for ground-truth labels that remains effective under distribution shift.

What carries the argument

Fused Reference Alignment Prediction (FRAP), which performs temperature-scaled alignment of foundation-model predictions to the base model followed by confidence-weighted fusion to produce a reference distribution used for agreement-based estimation.

Load-bearing premise

Temperature-scaled alignment followed by confidence-weighted fusion produces a reference distribution whose agreement with base-model outputs faithfully tracks true accuracy without ground-truth labels on the shifted test set.

What would settle it

On a shifted test set with held-out ground-truth labels, measure actual accuracy and check whether FRAP's agreement-based estimate shows low or zero correlation with it.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • FRAP yields consistent and substantial improvements over representative performance-estimation methods under distribution shift.
  • The method requires no ground-truth labels on the unlabeled shifted test set.
  • It integrates complementary strengths so that the reference distribution reflects both generalization and domain expertise.
  • Estimation reduces to measuring agreement between base-model outputs and the fused reference.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If stronger foundation models become available, the same alignment-and-fusion steps could be reused to produce still tighter estimates.
  • The construction of an internal reference distribution could be adapted to other unsupervised evaluation settings such as regression or structured prediction.
  • The temperature-scaling step might be replaced by other calibration techniques while preserving the overall fusion logic.
  • Repeated application of FRAP across successive shifts could serve as a monitor for gradual degradation in deployed systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes Fused Reference Alignment Prediction (FRAP) for estimating a base model's performance on an unlabeled test set under distribution shift. FRAP aligns an external foundation model's prediction distribution to the base model via temperature scaling that minimizes divergence, fuses the aligned outputs through confidence-based weighting to form a refined reference distribution, and obtains the performance estimate from the agreement between the base model's predictions and this reference. The authors report that extensive experiments across diverse datasets and architectures demonstrate consistent and substantial improvements over representative performance-estimation baselines.

Significance. If the central claim is substantiated, FRAP would offer a practical approach to performance estimation without ground-truth labels by exploiting complementary strengths of foundation models and domain-specific base models. This addresses a recognized limitation of methods that rely solely on the base model's outputs, whose biases increase under shift. The approach is notable for its use of an external model to construct a surrogate label distribution rather than direct calibration of the base model alone.

major comments (2)
  1. [Method] Method section (FRAP alignment and fusion steps): the central claim that temperature-scaled alignment minimizing divergence plus confidence-weighted fusion yields a reference distribution whose agreement with the base model faithfully proxies true accuracy is load-bearing, yet the manuscript provides no analysis or conditions under which this holds when the foundation and base models exhibit correlated errors under shift. Shared failure modes on out-of-domain features would render the surrogate no better than the base model alone, directly undermining the reported gains.
  2. [Experiments] Experimental evaluation: the abstract asserts 'consistent and substantial improvements' but the manuscript does not report quantitative results, error bars, dataset characteristics, or ablation studies isolating the contribution of the alignment versus fusion steps. Without these, it is impossible to assess whether the gains are robust or whether the method merely benefits from the foundation model's stronger out-of-domain performance in the chosen test cases.
minor comments (2)
  1. [Abstract] The abstract and method description use the term 'complementary strengths' without defining the precise conditions (e.g., error correlation thresholds) under which complementarity is expected to hold.
  2. [Method] Notation for the temperature scaling parameter and the confidence weighting function should be introduced with explicit equations rather than descriptive text only.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the major comments point-by-point below and commit to revisions that strengthen the manuscript.

read point-by-point responses

  1. Referee: [Method] Method section (FRAP alignment and fusion steps): the central claim that temperature-scaled alignment minimizing divergence plus confidence-weighted fusion yields a reference distribution whose agreement with the base model faithfully proxies true accuracy is load-bearing, yet the manuscript provides no analysis or conditions under which this holds when the foundation and base models exhibit correlated errors under shift. Shared failure modes on out-of-domain features would render the surrogate no better than the base model alone, directly undermining the reported gains.

    Authors: We agree that the current manuscript lacks an explicit analysis of robustness to correlated errors between the foundation and base models. The confidence-weighted fusion is intended to mitigate this by down-weighting the foundation model in low-confidence regions (often where shared failures occur), but we acknowledge this is not formally characterized. We will add a new subsection on limitations and failure modes, along with controlled synthetic experiments that vary the degree of error correlation to delineate when FRAP improves upon the base model alone. revision: yes

  2. Referee: [Experiments] Experimental evaluation: the abstract asserts 'consistent and substantial improvements' but the manuscript does not report quantitative results, error bars, dataset characteristics, or ablation studies isolating the contribution of the alignment versus fusion steps. Without these, it is impossible to assess whether the gains are robust or whether the method merely benefits from the foundation model's stronger out-of-domain performance in the chosen test cases.

    Authors: Section 4 and the appendix already contain tables with quantitative estimates, standard-error bars over multiple runs, dataset descriptions (including shift types and sizes), and ablations separating alignment from fusion. However, these elements are not sufficiently foregrounded. We will revise the abstract to cite specific average improvements, expand the main-text dataset summary, and add a dedicated main-text ablation isolating each component's contribution. revision: partial

Circularity Check

0 steps flagged

No circularity; FRAP is an independent algorithmic construction

full rationale

The paper presents FRAP as a new procedure: temperature-scaled alignment of an external foundation model to the base model (minimizing divergence), confidence-weighted fusion into a reference distribution, and performance estimation via agreement between base-model outputs and that reference. No equations or steps reduce by construction to the inputs; the foundation model is treated as external and independent. No self-citations, uniqueness theorems, or fitted parameters are invoked as load-bearing derivations. The abstract and description contain only method definition plus empirical claims, making the derivation self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

Review based solely on abstract; full details on parameters and assumptions unavailable. Temperature scaling and confidence weighting appear as tunable elements whose values are not specified.

free parameters (2)
  • temperature scaling parameter
    Used to align prediction distributions between foundation and base models.
  • confidence weighting threshold or function
    Used to fuse aligned predictions into the reference distribution.
axioms (1)
  • domain assumption An external foundation model supplies robustness that complements the base model's domain-specific predictions under distribution shift.
    Invoked to justify the fusion step as producing a reliable surrogate.

pith-pipeline@v0.9.1-grok · 5721 in / 1036 out tokens · 25366 ms · 2026-06-28T01:58:01.536341+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Bridging Domain Expertise and Generalization for Performance Estimation." pith.science (2026). https://pith.science/paper/2CDA33RA

@misc{pith2026260606335,
  author       = {Pith},
  title        = {Pith review of: Bridging Domain Expertise and Generalization for Performance Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CDA33RA}},
  note         = {Machine review of arXiv:2606.06335}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Performance estimation under distribution shift aims to predict how a model behaves on an unlabeled test set whose distribution differs from the training data, a scenario that requires reliable indicators that can faithfully reflect model behavior without ground-truth labels. Existing approaches rely solely on the outputs of the given model whose biases are amplified once the distribution shifts, weakening the correlation with the true performance. Motivated by this limitation, we propose Fused Reference Alignment Prediction (FRAP), which leverages the complementary strengths of an external foundation model and the base model to construct a more reliable surrogate of the ground-truth labels. FRAP aligns the prediction distribution of the foundation model with that of the base model by applying temperature-scaled calibration that minimizes their divergence. The aligned predictions are fused through confidence-based weighting into a refined reference distribution that integrates robustness from the foundation model and domain-specific expertise from the base model, and performance estimation is obtained by measuring how closely the base model predictions agree with this reference. Extensive experiments across diverse datasets and architectures show that FRAP provides consistent and substantial improvements over representative performance-estimation methods under distribution shift.

Figures

Figures reproduced from arXiv: 2606.06335 by Quyu Kong, Shuxuan Li, Wei-Shi Zheng, Zhilin Zhao.

Figure 1
Figure 1. Figure 1: FRAP Overview. FRAP framework leverages a source-trained base model fθ together with a robust foundation model fr. The prediction from the foundation model is first calibrated via temperature scaling τ guided by the output of the base model during test time, and is subsequently fused with the base model prediction through a confidence-weighting scheme. The resulting refined predictive distribution acts as … view at source ↗
Figure 2
Figure 2. Figure 2: Calibration behavior of raw CLIP similarity scores across diverse datasets. The green dashed line indicates the ran￾dom baseline corresponding to uniform prediction over the label space (nearly overlapping with the red confidence curve). and is converted into mixture weights over the two models by wm(x) = cm(x) P m′∈{b,r} cm′ (x) , m ∈ {b, r}. In particular, the weight of the base model is α(x) = wb(x) and… view at source ↗
Figure 3
Figure 3. Figure 3: Fixed temperature vs. Test-time calibration. We com￾pare the ECE of our test-time-learned temperature (TTC) against the fixed temperature values (τ = 0.1, 0.05, 0.01) across diverse datasets, with CLIP serving as the foundation model. sured by ECE, does not necessarily translate into improved estimation performance. We attribute this discrepancy to a fundamental mismatch between the calibration objective a… view at source ↗
Figure 4
Figure 4. Figure 4: Effectiveness of CWF. Per-experiment Semantic Alignment Score (SAS) gaps on ImageNet. We compare the confidence-weighted fusion against the base model (left column) and CLIP (right column) under (a) natural shifts and (b) synthetic corruptions. Positive similarity gap ∆SAS = SASfusion − SASbaseline indicate improved semantic alignment (∆SAS > 0) and the fusion predominantly achieves positive differences, v… view at source ↗
Figure 5
Figure 5. Figure 5: Per-experiment Semantic Alignment Score (SAS) differences on Living-17. We compare the confidence-weighted fusion against the base model (left column) and CLIP (right column) under (a) natural shifts and (b) synthetic corruptions. The y-axis shows ∆SAS = SASfusion − SASbaseline. Positive values indicate improved semantic alignment (∆SAS > 0), while negative values indicate degradation. The fusion largely e… view at source ↗
Figure 6
Figure 6. Figure 6: Per-experiment Semantic Alignment Score (SAS) differences on Nonliving-26. (a) same shifts and (b) novel shifts is the same with the Living-17. The fusion consistently achieves better semantic alignment than both the base model and CLIP. 0 20 40 60 80 Test Sample Index 0.2 0.0 0.2 0.4 0.6 Similarity Gap Improvement: 30.0% Mean: -0.081 Fusion vs. Base Model 0 20 40 60 80 Test Sample Index Improvement: 90.0%… view at source ↗
Figure 7
Figure 7. Figure 7: Per-experiment Semantic Alignment Score (SAS) differences on Tiny-ImageNet. (a) natural shifts and (b) synthetic corruptions is similar to the ImageNet. While the fused distribution on this dataset provides limited gains over the base model. with the observed error rate of the base model on Ds. This process stabilizes the estimation by correcting the scale and offset between the raw prediction score and th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 4 canonical work pages · 4 internal anchors

  1. [1]

    Zico Kolter

    Christina Baek, Yiding Jiang, Aditi Raghunathan, and J. Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. InAdvances in Neural Information Processing Systems (NeurIPS), pages 19274–19289, 2022. 1, 2

    2022

  2. [2]

    On the Opportunities and Risks of Foundation Models

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castel- lon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Es...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  3. [3]

    Chen, Karan Goel, Nimit Sharad Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher R´e

    Mayee F. Chen, Karan Goel, Nimit Sharad Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher R´e. Mandoline: Model evaluation under distribution shift. InInternational Confer- ence on Machine Learning (ICML), pages 1617–1629, 2021. 5, 3

    2021

  4. [4]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geof- frey E. Hinton. A simple framework for contrastive learning of visual representations. InInternational Conference on Machine Learning (ICML), pages 1597–1607, 2020. 1

    2020

  5. [5]

    Christie, Neil Fendley, James Wilson, and Ryan Mukherjee

    Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 6172–6180, 2018. 5, 2

    2018

  6. [6]

    Confidence and dispersity speak: Characterising prediction matrix for unsupervised accuracy estimation

    Weijian Deng, Yumin Suh, Stephen Gould, and Liang Zheng. Confidence and dispersity speak: Characterising prediction matrix for unsupervised accuracy estimation. InInternational Conference on Machine Learning (ICML), pages 7658–7674,

  7. [7]

    Barrett, Yi Xiang, Miguel Romero Calvo, Anna Currey, and Xing Niu

    Georgiana Dinu, Corey D. Barrett, Yi Xiang, Miguel Romero Calvo, Anna Currey, and Xing Niu. Effective post-training embedding compression via temperature control in contrastive training. InInternational Conference on Learning Represen- tations (ICLR), pages 1–10, 2025. 4, 2

    2025

  8. [8]

    Data determines distributional robustness in contrastive language image pre-training (CLIP)

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). InInternational Conference on Machine Learning (ICML), pages 6216–6234, 2022. 3

    2022

  9. [9]

    Lempitsky

    Yaroslav Ganin and Victor S. Lempitsky. Unsupervised do- main adaptation by backpropagation. InInternational Confer- ence on Machine Learning (ICML), pages 1180–1189, 2015. 2

    2015

  10. [10]

    Leveraging unla- beled data to predict out-of-distribution performance

    Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lip- ton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unla- beled data to predict out-of-distribution performance. InIn- ternational Conference on Learning Representations (ICLR), pages 1–10, 2022. 1, 5, 7, 4

    2022

  11. [11]

    Predicting with confidence on unseen distributions

    Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 1114–1124, 2021. 1, 2, 5, 3

    2021

  12. [12]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InInternational Conference on Machine Learning (ICML), pages 1321–1330,

  13. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5

    2016

  14. [14]

    Sample selection bias as a specification error.Econometrica, 47(1):153–161, 1979

    James J Heckman. Sample selection bias as a specification error.Econometrica, 47(1):153–161, 1979. 2

    1979

  15. [15]

    A baseline for detect- ing misclassified and out-of-distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. InInternational Conference on Learning Repre- sentations (ICLR), pages 1–9, 2017. 3

    2017

  16. [16]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 8320–8329, 2021. 3

    2021

  17. [17]

    Weinberger

    Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kil- ian Q. Weinberger. Densely connected convolutional net- works. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017. 5

    2017

  18. [18]

    Jonathan J. Hull. A database for handwritten text recognition research.IEEE Trans. Pattern Anal. Mach. Intell., 16(5): 550–554, 1994. 3

    1994

  19. [19]

    Zico Kolter

    Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J. Zico Kolter. Assessing generalization of SGD via dis- agreement. InInternational Conference on Learning Repre- sentations (ICLR), pages 1–9, 2022. 1, 2, 5, 3

    2022

  20. [20]

    Domain adaptation without source data.IEEE Trans

    Youngeun Kim, Donghyeon Cho, Kyeongtak Han, Priyadarshini Panda, and Sungeun Hong. Domain adaptation without source data.IEEE Trans. Artif. Intell., 2(6):508–518,

  21. [21]

    Haque, Sara M

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran S. Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS...

    2021

  22. [22]

    Learning multiple lay- ers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple lay- ers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009. 5, 2

    2009

  23. [23]

    V ., and R

    Jogendra Nath Kundu, Naveen Venkat, Rahul M. V ., and R. Venkatesh Babu. Universal source-free domain adaptation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4543–4552, 2020. 2

    2020

  24. [24]

    Gradient-based learning applied to document recog- nition.Proc

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition.Proc. IEEE, 86(11):2278–2324, 1998. 5, 2

    1998

  25. [25]

    Confidence score for source-free unsupervised domain adap- tation

    Jonghyun Lee, Dahuin Jung, Junho Yim, and Sungroh Yoon. Confidence score for source-free unsupervised domain adap- tation. InInternational Conference on Machine Learning (ICML), pages 12365–12377, 2022. 2

    2022

  26. [26]

    Model adaptation: Unsupervised domain adaptation without source data

    Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9638–9647,

  27. [27]

    Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation

    Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. InInternational Conference on Machine Learning (ICML), pages 6028–6039, 2020. 2

    2020

  28. [28]

    Divergence measures based on the shannon entropy.IEEE Trans

    Jianhua Lin. Divergence measures based on the shannon entropy.IEEE Trans. Inf. Theory, 37(1):145–151, 1991. 2

    1991

  29. [29]

    Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. InInternational Conference on Machine Learning (ICML), pages 97–105, 2015. 2

    2015

  30. [30]

    Harder or different? a closer look at distribution shift in dataset reproduction

    Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todes- chini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? a closer look at distribution shift in dataset reproduction. InICML Workshop on Uncertainty and Robustness in Deep Learning, pages 1–15, 2020. 3

    2020

  31. [31]

    Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ke- tong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepput- tis, Joseph Campbell, and Katia P. Sycara. Characterizing out-of-distribution error via optimal transport. InAdvances in Neural Information Processing Systems (NeurIPS), pages 17602–17622, 2023. 1, 2, 5, 4

    2023

  32. [32]

    George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, 1995. 7

    1995

  33. [33]

    Revisiting the calibration of modern neural net- works

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural net- works. InAdvances in Neural Information Processing Systems (NeurIPS), pages 15682–15694, 2021. 4

    2021

  34. [34]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning, pages 1–8, 2011. 3

    2011

  35. [35]

    A survey on transfer learn- ing.IEEE Trans

    Sinno Jialin Pan and Qiang Yang. A survey on transfer learn- ing.IEEE Trans. Knowl. Data Eng., 22(10):1345–1359, 2010. 3

    2010

  36. [36]

    The entropy enigma: Success and failure of en- tropy minimization

    Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of en- tropy minimization. InInternational Conference on Machine Learning (ICML), pages 41064–41085, 2024. 1, 2

    2024

  37. [37]

    Development and application of a metric on semantic nets

    Roy Rada, Hafedh Mili, Ellen Bicknell, and Maria Blettner. Development and application of a metric on semantic nets. IEEE Trans. Syst. Man Cybern., 19(1):17–30, 1989. 7

    1989

  38. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICML), pages 8748–8763,

  39. [39]

    Do CIFAR-10 Classifiers Generalize to CIFAR-10?

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10?CoRR, abs/1806.00451:1–13, 2018. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  40. [40]

    Do imagenet classifiers generalize to im- agenet? InInternational Conference on Machine Learning (ICML), pages 5389–5400, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? InInternational Conference on Machine Learning (ICML), pages 5389–5400, 2019. 1, 2

    2019

  41. [41]

    Can contrastive learning avoid shortcut solutions? InAdvances in Neural Information Processing Systems (NeurIPS), pages 4974–4986, 2021

    Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? InAdvances in Neural Information Processing Systems (NeurIPS), pages 4974–4986, 2021. 1

    2021

  42. [42]

    Distance matters for improving performance estimation under covariate shift

    M´elanie Roschewitz and Ben Glocker. Distance matters for improving performance estimation under covariate shift. InIEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 4551–4561, 2023. 1

    2023

  43. [43]

    (almost) provable er- ror bounds under distribution shift via disagreement discrep- ancy

    Elan Rosenfeld and Saurabh Garg. (almost) provable er- ror bounds under distribution shift via disagreement discrep- ancy. InAdvances in Neural Information Processing Systems (NeurIPS), pages 28761–28784, 2023. 1, 2

    2023

  44. [44]

    Bernstein, Alexander C

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211–252, 2015. 5, 2

    2015

  45. [45]

    BREEDS: benchmarks for subpopulation shift

    Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS: benchmarks for subpopulation shift. InInterna- tional Conference on Learning Representations (ICLR), pages 1–9, 2021. 5, 2

    2021

  46. [46]

    Hoffmann, Max Argus, V olker Fis- cher, and Thomas Brox

    Simon Schrodi, David T. Hoffmann, Max Argus, V olker Fis- cher, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in con- trastive vision-language models. InInternational Conference on Learning Representations (ICLR), pages 1–10, 2025. 4, 2

    2025

  47. [47]

    Improving predictive inference under covariate shift by weighting the log-likelihood function.J

    Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function.J. Stat. Plan. Inference, 90(2):227–244, 2000. 2

    2000

  48. [48]

    Sculley, Joshua V

    Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshmi- narayanan, Sebastian Nowozin, D. Sculley, Joshua V . Dillon, Jie Ren, and Zachary Nado. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. InAdvances in Neural Information Processing Systems (NeurIPS), pages 13969–13980, 2019. 1, 4

    2019

  49. [49]

    Proxy denoising for source-free domain adaptation

    Song Tang, Wenxin Su, Yan Gan, Mao Ye, Jianwei Zhang, and Xiatian Zhu. Proxy denoising for source-free domain adaptation. InInternational Conference on Learning Repre- sentations (ICLR), pages 1–10, 2025. 2

    2025

  50. [50]

    A closer look at the robustness of contrastive language-image pre-training (CLIP)

    Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (CLIP). InAdvances in Neural Information Processing Sys- tems (NeurIPS), pages 13678–13691, 2023. 3

    2023

  51. [51]

    Deep Domain Confusion: Maximizing for Domain Invariance

    Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance.CoRR, abs/1412.3474:1–8, 2014. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  52. [52]

    Adversarial discriminative domain adaptation

    Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2962–2971, 2017. 2

    2017

  53. [53]

    Representation Learning with Contrastive Predictive Coding

    A¨aron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.CoRR, abs/1807.03748:1–9, 2018. 1

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  54. [54]

    Lipton, and Eric P

    Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global representations by penalizing local predictive power. InAdvances in Neural Information Processing Systems (NeurIPS), pages 10506–10518, 2019. 3

    2019

  55. [55]

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere

    Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. InInternational Conference on Machine Learning (ICML), pages 9929–9939, 2020. 4, 1, 2

    2020

  56. [56]

    Verb semantics and lexical selection

    Zhibiao Wu and Martha Stone Palmer. Verb semantics and lexical selection. InAssociation for Computational Linguis- tics (ACL), pages 133–138, 1994. 7

    1994

  57. [57]

    Hovy, and Quoc V

    Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V . Le. Self-training with noisy student improves imagenet clas- sification. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2020. 2

    2020

  58. [58]

    On the importance of feature separability in predicting out-of-distribution error

    Renchunzi Xie, Hongxin Wei, Lei Feng, Yuzhou Cao, and Bo An. On the importance of feature separability in predicting out-of-distribution error. InAdvances in Neural Information Processing Systems (NeurIPS), pages 27783–27800, 2023. 1, 2

    2023

  59. [59]

    Cold case: The lost MNIST digits

    Chhavi Yadav and L´eon Bottou. Cold case: The lost MNIST digits. InAdvances in Neural Information Processing Systems (NeurIPS), pages 13443–13452, 2019. 3

    2019

  60. [60]

    How transferable are features in deep neural networks? InAd- vances in Neural Information Processing Systems (NeurIPS), pages 3320–3328, 2014

    Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? InAd- vances in Neural Information Processing Systems (NeurIPS), pages 3320–3328, 2014. 3

    2014

  61. [61]

    Predicting out-of-distribution error with the projection norm

    Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Ja- cob Steinhardt. Predicting out-of-distribution error with the projection norm. InInternational Conference on Machine Learning (ICML), pages 25721–25746, 2022. 1, 2, 5, 4

    2022

  62. [62]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941– 11952, 2023. 3

    2023

  63. [63]

    Yang Zou, Zhiding Yu, B. V . K. Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic seg- mentation via class-balanced self-training. InEuropean Con- ference on Computer Vision (ECCV), pages 297–313, 2018. 2 Bridging Domain Expertise and Generalization for Performance Estimation Supplementary Material A. Derivation of Proposition 1 ...

    2018

  64. [64]

    Average Confidence (AC).THe AC baseline directly esti- mates the target error by computing the average of one minus the maximum softmax confidence over the unlabeled target samples

    this effectively corresponds to using a single slice in the classifier confidence space. Average Confidence (AC).THe AC baseline directly esti- mates the target error by computing the average of one minus the maximum softmax confidence over the unlabeled target samples. Difference of Confidence (DoC).It is also known as DOC- Feat, which models the error a...